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1.  Introduction 


A  model  is  a  simplified  representation  of  the  world  that  contains  the  essential  elements  of  effective 
processes  for  a  specific  body  of  knowledge.  CMM®s  have  been  developed  for  organizations  that  use 
specialized  bodies  of  knowledge  to  improve  their  internal  process  maturity,  as  well  as  to  evaluate 
their  processes.  CMMs  are  being  used  in  several  intellectual  endeavors  to  include  software 
engineering,  Software  Acquisition  (SA),  and  systems  engineering.  Four  of  these  models  have  been 
integrated  together  into  the  Software  Engineering  Institute’s  (SEI’s)  Capability  Maturity  Model  Inte¬ 
gration  (CMMIsm).  This  model  integrates  Systems  Engineering  (SE),  Software  Engineering  (SW), 
Integrated  Product  and  Process  Development  (EPPD),  and  Supplier  Sourcing  (SS).  Together  these 
have  the  acronym  CMMI-SE/SW/IPPD/SS.1 

Our  research  has  not  found  a  CMM  for  the  processes  used  by  analysis  organizations.  We  have  cre¬ 
ated  the  Analysis  Capability  Maturity  Model  (ACMM)  to  provide  a  process  framework  for  the  body 
of  knowledge  related  to  analysis  and  operations  research."  The  CMM  concepts  were  used  in  creation 
of  this  model,  which  is  also  founded  on  best  practices  from  the  Intelligence  Community,  DoD,  aca¬ 
demia,  and  industry .2’5'7-8,10,15  Readers  familiar  with  the  CMMI  will  note  that  the  Generic  Goals  and 
Practices  were  directly  incorporated.  Several  process  areas  were  adapted  from  the  CMMI. 

The  ACMM  identifies  five  levels  of  maturity.  These  levels  correspond  to  the  CMMI  levels  and  are 
defined  as  follows: 

1 .  The  initial  level,  characterized  by  ad  hoc  processes,  which  can  be  chaotic. 

2.  The  managed  level,  characterized  by  managed  analysis  processes,  work  products,  and 
services  for  individual  studies. 

3.  The  defined  level,  characterized  by  standard  analysis  processes,  work  products,  and 
services  for  the  analysis  organization. 

4.  The  quantitative  level,  characterized  by  analysis  tasks  being  managed  quantitatively. 

5.  The  optimizing  level,  with  continuous  improvement  of  analytic  processes. 

This  analysis  capability  maturity  model  can  be  used  as  guidance  for  improving  an  analysis  organiza¬ 
tion’s  processes  and  its  ability  to  manage  the  development,  acquisition,  and  maintenance  of  analysis 
products  and  services.  The  ACMM  can  also  be  used  to  help  set  process  improvement  objectives  and 
priorities,  improve  processes,  and  provide  guidance  for  ensuring  stable,  capable,  and  mature 
processes. 


*  CMM  and  Capability  and  Maturity  Model  are  registered  in  the  U.S.  Patent  and  Trademark  Office. 

SM  CMMI  is  a  service  mark  of  Camegie-Mellon  University. 

'  Morse  and  Kimball  define  Operations  Research  as:  “  . .  .a  scientific  method  of  providing  executive  departments  with  a 
quantitative  basis  for  decisions  regarding  operations  under  their  control.” 
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The  ACMM  does  not  specify  the  implementation  necessary  for  any  particular  process  area.  Process 
development  depends  on  the  analysis  organization’s  unique  needs  and  is  their  responsibility.  The 
ACMM  does  not  establish  goodness  criteria  for  each  process.  The  focus  is  on  the  existence  of  proc¬ 
esses  and  their  management.  Development  of  effective  analysis  processes  depends  on  implementing 
the  generic  practices  associated  with  each  specific  practice.  Generic  practices  provide  institutionali¬ 
zation  to  ensure  that  the  associated  processes  will  be  effective,  repeatable,  and  lasting.  The  generic 
goals  are  characterized  by  four  common  features:  those  designed  to  produce  an  organizational  com¬ 
mitment  to  perform  the  specific  practice;  those  designed  to  ensure  the  ability  to  perform  the  specific 
practice;  those  directing  implementation;  and  those  to  verify  implementation. 

If  the  ACMM  goals  are  satisfied,  there  are  potential  benefits  to  the  organization’s  ability  to  perform 
quality  analyses,  resulting  in  mission  success  for  the  analysis  organization.  Following  well-defined, 
effective  analysis  processes  promotes  customer  satisfaction  and  improves  resource  efficiency,  analy¬ 
sis  quality,  and  timeliness.  At  higher  maturity  levels,  analyses  will  be  delivered  on  schedule  and 
budget.  The  ACMM  is  unique  in  that  processes  areas  are  provided  to  forecast  and  develop  the  neces¬ 
sary  skills  and  capabilities  for  future  studies. 

The  ACMM  can  be  used  as  an  appraisal  instrument  for  exploratory  appraisals,  which  are  conducted  to 
give  management  an  assessment  of  the  maturity  of  their  analysis  processes  within  the  organization.  It 
can  also  be  used  as  a  verification  appraisal  instrument  to  confirm  an  organization’s  maturity  level 
following  process  improvement  activities.  An  appraisal  can  be  used  to  qualify  or  select  an  analysis 
organization  to  perform  an  important  study. 

An  independent  team,  which  did  not  include  the  authors  of  the  ACMM,  used  an  initial  version  of  the 
ACMM  to  perform  a  confidential  appraisal  of  a  system  engineering  and  analysis  organization.  The 
appraisal  evaluated  the  analysis  organization’s  capability  to  produce  high-quality  analysis  products 
and  services,  focusing  on  current  analysis  work  and  comparing  this  work  with  the  analysis  needed  to 
support  the  organization’s  mission.  The  appraisal  recommended  changes  to  the  processes  being  used, 
identified  tool  shortfalls,  and  recommended  development  projects  to  close  perceived  analysis  defi¬ 
ciencies.  The  team  recommended  minor  changes  to  the  ACMM,  which  have  been  incorporated  in  this 
document.  Section  7,  Analysis  Capability  Appraisal  draws  on  the  experience  of  this  team.8, 9 

The  ACMM  is  recommended  to  analysis  organizations  interested  in  improving  the  reliability  and 
quality  of  their  studies.  It  is  also  recommended  to  the  purchasers  of  operations  research  and  system 
analysis  who  are  often  asked  to  take  the  conclusions  and  recommendations  of  a  study  on  faith  alone. 
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2.  ACMM  Creation 


Creation  of  the  ACMM  began  in  order  to  facilitate  an  independent  appraisal  of  analysis  within  a  gov¬ 
ernment  program.  The  first  step  for  the  independent  appraisal  was  to  create  a  brief  statement  of  work 
or  charter  for  the  effort.  A  search  for  a  lead  assessor  was  begun  shortly  thereafter.  During  this 
search,  it  became  apparent  that  the  charter  needed  elaboration,  especially  with  regard  to  the  appraisal 
of  analysis  processes. 

A  literature  search  revealed  some  interest  in  the  processes  to  conduct  some  phases  of  analyses.  How¬ 
ever,  a  holistic  treatment  of  analytic  organizational  process  was  not  identified.  The  formal  academic 
training  for  operations  researchers  apparently  has  a  principal  focus  on  optimization  techniques. 
Problem  formulation  and  metrics  development  are  also  taught.5  It  appears  that  the  operations 
research  community  depends  upon  on-the-job  training  for  study  organization,  execution,  and  man¬ 
agement.  The  development  and  sustainment  of  analysis  capabilities  within  the  organization  was  a 
major  concern  for  the  appraisal,  yet  there  was  little  how-to  information  on  this  topic.  Several  inter¬ 
esting  analysis  organization  histories  were  found,3’4  but  these  contained  little  about  process. 

Members  of  the  lead-assessor  search  team  had  experience  with  Operations  Research  and  exposure  to 
Capability  Maturity  Models.  Based  on  the  Operations  Research  experience,  a  study  model  was  con¬ 
structed  centered  on  the  phases  of  analysis.  This  model  was  compared  to  the  CMMI;  significant 
intersection  between  the  two  models  was  observed.  However,  there  were  also  significant  differences. 
We  decided  to  use  CMM  concepts.  Appropriate  process  areas  from  the  CMMI  were  combined  with 
process  areas  created  for  unique  analysis  processes.  The  result  was  the  Analysis  Capability  Maturity 
Model  (ACMM). 

Figure  2. 1  provides  an  overview  of  the  study  process  model  used  in  developing  the  ACMM.  Reviews 
with  the  customer  are  called  for  in  several  phases,  which  are  opportunities  for  process  iteration. 

We  decided  to  specify  five  levels  of  maturity  for  the  ACMM  that  parallel  those  of  other  CMMs.  The 
maturity  levels  identification  serve  as  targets  for  process  improvement  for  any  analysis  organization; 
as  well  as,  managing  expectations  for  a  discovery  appraisal.  Within  each  process  area  are  Specific 
Goals  (SG)  and  Specific  Practices  (SP).  For  the  initial  appraisal,  SG  and  SP  for  level  three  and  below 
were  evaluated.  Levels  four  and  five  were  created  for  model  completeness. 

Associated  with  each  Specific  Practice  of  the  CMMI  are  Generic  Goals  (GG)  and  Generic  Practices 
(GP).  The  GG  and  GP  focus  on  institutionalizing  that  Specific  Practice.  Both  the  SP  and  the  GP  are 
needed.  For  the  ACMM,  the  Generic  Goals  and  Generic  Practices  were  adopted  from  the  CMMI. 

Several  of  the  SG  and  SP  were  also  adopted  from  the  CMMI.  Others  were  adapted  to  the  ACMM.  In 
some  cases,  the  concepts  are  the  same  but  a  different  lexicon  is  used.  Because  the  ACMM  deviates 
substantially  from  the  CMMI,  responsibility  for  the  model  rests  entirely  with  the  authors. 


3. 


Decision  Questions  , 


&  Issues 


.onsumcr/ 

Decision 

Makers 


|C  oninmnications 
&  Feedback  X 


Quantitative  Basis  For  Decisions 

["Establish  &  Maintain  Analysis  Capability 


Analysis  Formulation 

1.  Identify  Issue 

2.  I dentifv  Decision 

3.  Identify  Stakeholders 
4  Spec  i  fy  I  evel  of  E  ffort 


Analysis  Preparation 

I  Develop  Analysis  Questions 

2.  Develop  deeision  criteria 

3.  Identify  mcommcnsurablcs 

4.  Specify  importance,  ranking,  of  criteria 

5.  Develop  measures 
6  Hound  the  analysis 
7.  Develop  alternatives 

X  Develop  ROMS  of  cost  schedule 
9.  Review  preparation  wilt)  consumer 


Analysis  Planning 

1 .  De\  elop  &  test  algorithms  to  eompute 
measures 

2.  Identify  eons! mints 

3.  Estimate  resources 

4.  I  sinuate  schedule 

5.  Develop  scenarios;  alternate  contexts  for 
decisions 

f>.  Test  scenarios  for  sufficiency,  detail.  &  issue 
illumination 

7.  Nominate  data  sources 
X.  Analysis  Plan  peer  review 

9.  He ' icw  aiiah si*  pi. omin^  with  the 

consumer 


Data  Planning  if  Real  World  Data 

1 .  Identify  data  sources 

2.  Plan  fur  data  collection 

3.  Develop  sampling  plan  experiments  design 

4  Plan  lor  data  aggregation'  reduction 

5  Plan  for  data  certification 

6.  Estimate  cost  of  data  collection 

7.  Rrvkw  emt  A  schedule  with  consumer 


Data  Planning  if  Expert  Solicitation 

1 .  Plan  for  data  elicitation 

2.  Develop  criteria  for  SMEs 

3.  Identify  candidate  SMF.s 

4.  Evaluate  candidate  SMI-  against  criteria 

5.  Select  SMEs 

6.  Estimate  SME  elicitation  cost 

7.  KoifM  cos<  &  schedule  with  consumer 


Data  Planning  if  Model  &  Simulation 

I  Develop  selection  criterion  for  M&S 

2.  Identify  M&S  alternatives 

3.  Select  M&S 

If  M&S  development  required 
a.  Estimate  development  cost  &  schedule 
h.  Review  with  appropriate  stakeholders 
c.  Modify  existing  M&S  or  create  new 
d  Develop  M&S 
e.  Monitor  dcNclopmcnt  progress 

4.  Plan  for  accreditation  of  M&S 

a.  Verification 
b  Validation 

5.  IXsign  run  plan  (sensitivity) 

6.  Develop  M&S  execution  cost  &  schedule 

7.  Rcxk-w  cost  &  schedule  with  consumer 


Analysis  Process 


Data  (fathering 

1 .  Data  collection 

2.  Monitor  &  control  data  collection  to 
assure  adherence  to  plan 

3.  Certify  data 

4  Data  delivery  lor  analysis 


Analysis  Execution 

1 .  Plot  the  data 

2.  Calculate  measures 


Analysis  Resolution 

1 .  Relate  analysis  to  issues  &  questions 

2.  Relate  measures  to  decision  criteria 

3.  l  ist  ineommcnsurahles  aiul  other  factors 


I  Analysis  Quality  Assurance  | 

'  7  — 


Analysis  Reporting 

1.  f  inal  decision  reporting 

2.  Lessons  learned 

3.  Analysis  details  documentation 

4.  Analysis  methods,  data.  &  results  archiving 

a.  Taxonomy,  data  model.  search  to 
facilitate  search 


Figure  2. 1 .  Analysis  process  flow. 


i 


i 


4 


i 


3.  Analysis/Study  Best  Practices 


3.1  Good  Analysis  Principles 

Through  many  years  of  practical  experience,  good  analysis  principles  (including  lessons  learned) 
have  been  developed  by  various  organizations  that  provide  valuable  guidance  for  any  analysis  organi¬ 
zation’s  use  to  avoid  repeating  past  problems.  Among  the  major  heuristics  that  good  analysis  organi¬ 
zations  follow  are: 

•  Use  qualified  expert  judgment. 

•  Choose  the  right  objectives. 

•  Test  for  sensitivity. 

•  Assess  alternative  designs  (as  important  as  analysis). 

•  Use  interdisciplinary  teams  (required,  not  optional). 

•  Do  not  emphasize  an  optimization  type  of  analysis  for  R&D  questions. 

•  Compare  multiple  contingencies  for  complex  or  enterprise  level  questions. 

•  Relevant  questions  with  partial  answers  are  better  than  non-relevant  questions  with  full 
answers. 

•  Cost  estimates  are  essential  for  choosing  among  alternatives. 

•  Good  ideas  are  worth  a  thousand  evaluations.10 


3.2  Successful  Analysis/Study  Processes 

The  following  processes  and  practices  have  been  developed  for  performing  successful  analyses  and 
studies  by  following  good  analysis  principles.  These  are  a  more  detailed  expansion  and  practical 
application  of  the  ideas  contained  in  the  good  analysis  principles  discussed  above.  (See  Figure  2.1) 


3.2.1  Formulate  the  Problem 

Initial  problem  formulation  is  the  responsibility  of  the  customer  of  the  study  or  analysis.  However, 
the  study  organization  is  often  called  upon  to  assist  the  customer  in  defining  the  problem.  In  the  ini¬ 
tial  iteration,  it  is  critical  to  begin  with  an  understanding  of  the  real  problem  rather  than  a  determina¬ 
tion  to  apply  readily  available  tools,  scenarios,  and  data.  Explicit  problem  formulation  must  precede 
construction  of  concepts  for  analysis  or  method  selection.  Consequently,  problem  formulation  should 
focus  on  what  needs  to  be  done  rather  than  on  what  can  be  done.  Proper  attention  to  problem  formu¬ 
lation  is  essential  to  the  overall  efficiency  and  quality  of  the  study.  An  understanding  of  the  decisions 
to  be  supported  by  the  analysis  and  the  viewpoints  of  the  various  stakeholders  helps  to  clarify  the 
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study  issues.7  The  following  ordered  steps  provide  a  solid,  proven  process  for  proper  formulation  of 
an  analysis  or  study: 

•  Prepare  problem  statement  (definition  of  decision/analysis  issues). 

•  Determine  why  the  study  is  being  done  (understand  how  analysis  influences  decisions). 

•  Work  with  the  customer  to  decide  on  customer’s  resource  investment  (level  of  effort). 

This  is  what  the  customer  is  willing  to  pay  to  have  quantitative  support  for  his  decision. 

•  Identify  all  of  the  stakeholders. 

•  Obtain  commitment  to  the  analysis  issues. 

•  Manage  analysis  issue  changes  (iterative  process). 

•  Maintain  consistency  between  analysis  results  and  issues  (iterative  process). 


3.2.2  Prepare  for  Analysis 

3.2.2. 1  Identify  Analysis  Decision  Factors 

Identifying  and  understanding  the  factors  that  affect  the  decision  process  is  vital  for  planning  and 
implementing  analyses.  Begin  with  the  end  in  mind.  The  following  factors  ensure  that  a  comprehen¬ 
sive  set  of  problem  solution  alternatives  are  analyzed  and  that  an  exit  strategy  has  been  defined  when 
considering  alternatives  so  that  excess  effort  is  not  applied  to  the  trade  space:2 

•  Develop  analysis  questions. 

•  Develop  decision  alternatives  and  criteria. 

•  Identify  any  constraints  and  assumptions. 

•  Develop  decision  criteria  and  rank  according  to  importance. 

•  Identify  incommensurables.  Often  in  a  decision,  the  incommensurables  are  more  signifi¬ 
cant  that  the  quantitative  factors. 

•  Bound  the  analysis.  A  tight  focus  on  the  really  important  elements  of  the  analysis  is 
needed  to  avoid  paralysis  by  analysis.  A  bounded  study  is  essential  to  estimate  costs  and 
schedule.  As  the  study  progresses,  bounding  the  analysis  initially  will  avoid  creep. 

•  Establish  rules  or  standards  for  alternatives. 

•  Develop  and  evaluate  alternatives. 

•  Select  an  alternative — this  implies  that  those  specific  resources  are  no  longer  available 
for  other  purposes 
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3.2.2.2  Develop  Quantitative  Measures  Using  Quantitative  Criteria 

Quantitative  criteria  must  be  applied  when  developing  measures  to  assess  mission  effectiveness  or 
functional  performance.  These  criteria  must  be  sensitive  to  the  independent  factors  of  the  analysis 
and  are  explored  using  sensitivity  analyses.  These  measures  will  also  be  the  means  for  arriving  at 
quantitative  results  that  can  be  used  as  the  basis  for  decision.  The  following  measures  help  to  deter¬ 
mine  when  the  analyses  are  successful: 

•  Measures  of  Effectiveness  (MoEs) — mission  performance  indicators 

•  Measure  of  Performance  (MoPs) — functional  performance  indicators 

•  Measures  of  Uncertainty  (MoU) — Uncertainty  is  permissible,  but  the  uncertainty  degree 
and  sources  must  be  documented.  Inserting  odds  ratios  or  numerical  probability  ranges  to 
clarify  key  points  should  be  standard  practice.6 

•  Measures  of  Cost — Cost  can  be  measured  in  a  number  of  ways:  acquisition  cost,  life- 
cycle  cost,  or  opportunity  cost.  A  consistent  cost  measure  is  needed  in  order  to  weigh 
cost  against  other  measures. 

•  Provide  means  for  weighing  cost  versus  performance. 

•  Develop  Rough  Orders  of  Magnitude  (ROMs)  of  cost/schedule. 

3.2.3  Prepare  an  Analysis  Plan 

Most  analysis  efforts  are  complex  in  nature  and  must  be  monitored  and  controlled  against  a  well 
thought  out  plan  of  action.  Technical  analysis  capability  as  well  as  consideration  of  limited  resources, 
tight  budgets,  tough  schedules,  and  associated  risk  management  must  be  accounted  for  in  the  plan. 
Adequate  time  must  be  allocated  for  collecting  data,  building  models,  developing  algorithms,  and 
preparing  operations  scenarios.  The  following  activities  can  help  in  the  development  of  a  good  analy¬ 
sis  plan: 


•  Specify  measures. 

•  Abstract  the  important  features  of  the  decision  or  problem. 

•  Establish  schedule  for  study  results. 

•  Identify  or  develop  algorithms  for  computing  the  measures. 

•  Specify  the  data  necessary  for  the  algorithm. 

•  Generate  alternatives,  justify  why  some  alternatives  are  deemed  less  likely,  and  express 
the  likelihood  that  events  may  not  turn  out  as  expected. 

•  Prepare  scenarios  (controlled  situations)  to  explore  the  trade  space — do  not  use  to  try  to 
define  requirements.  Scenarios  provide  a  framework  for  exploring  the  implications  of 
decisions. 
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The  products  of  these  activities  should  be  included  in  the  plan. 

3.2.4  Plan  for  Data 

There  are  three  likely  sources  for  analysis  data:  the  real  world,  either  data  that  is  gathered  from 
operations,  exercises,  or  experiments;  models  and  simulations;  and  subject  matter  experts.  Each  of 
these  data  sources  needs  to  be  identified  and  qualified  in  some  fashion.  The  following  activities 
should  be  used  for  the  development  of  the  data  plan: 

•  Identify  data  sources. 

•  Plan  for  data  collection/elicitation. 

•  Develop  data  sampling  plan. 

•  If  using  SMEs,  develop  selection  criteria  and  identify  candidates. 

•  Evaluate  SME  candidates  against  criteria  and  make  selections. 

•  If  using  models  and  simulations,  develop  selection  criteria. 

•  Identify  appropriate  models  and  simulations. 

•  Identify  Modeling  and  Simulation  (M&S)  alternatives  and  make  selection. 

•  Identify  required  M&S  development  requirements. 

•  If  development  required,  estimate  cost  and  schedule. 

•  Review  with  stakeholders. 

•  Modify  existing  M&S  or  create  new. 

•  Monitor  development  progress. 

•  Design  run  plan  (sensitivity). 

•  Plan  for  data  aggregation/reduction. 

•  Plan  for  data  certification/M&S  accreditation. 

•  Estimate  cost  and  schedule  of  data  planning. 

The  data  plan  should  be  reviewed  for  completeness.  There  is  nothing  more  frustrating  than  to  dis¬ 
cover  missing  elements  that  are  essential  to  the  analysis.  It  is  good  practice  to  conduct  a  rehearsal  if 
data  is  to  be  gathered  in  the  real  world.  For  models  and  simulations,  a  small  pilot  may  reveal  any 
flaws  in  the  data  planning. 

3.2.5  Gather  Data 

After  all  the  planning  and  preparation  is  completed,  the  data  is  gathered  and  delivered  to  the  analyst 
for  processing  into  the  study  products.  The  data  collection  must  be  monitored  and  controlled  to 
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assure  adherence  to  the  data  plan.  The  data  must  be  certified  to  assure  credible  analysis  results  and 
then  delivered  to  the  analysts  in  a  timely  manner  to  maintain  schedules. 


3.2.6  Execute  the  Analysis  Plan 

Once  the  plan  has  been  successfully  developed  and  approved  by  the  decision  makers,  it  must  be  exe¬ 
cuted  according  to  the  activities,  resources,  and  schedules  in  the  plan.  It  becomes  the  living  document 
to  monitor  and  control  progress  of  the  analysis  effort.  When  decision  needs  change,  as  they  fre¬ 
quently  do,  the  plan  must  be  flexible  enough  to  allow  modification  without  major  interruption  to  the 
process.  The  following  activities  are  vital  for  effective  management  and  implementation  of  planning 
and  for  producing  quality  analyses: 

•  Monitor  plan  execution. 

•  Identify  data  shortfalls  and  fill  the  gaps  where  data  is  missing. 

•  Monitor  progress  in  M&S  modification. 

•  Test  the  scenarios  and  algorithms.  Assess  the  impact  of  the  proposed  capability  on  the 
problem  situation. 

•  Document  and  communicate  the  results. 

3.2. 6.1  Analysis  Resolution  &  QA 

The  data  produced  by  the  analysis  needs  to  be  interpreted  to  find  meaning.  In  addition,  peer  review 
and  critical  examination  is  needed  to  avoid  errors.  Providing  and  maintaining  issues  and  analysis 
results  traceability,  relating  measures  to  decision  criteria,  and  listing  incommensurables  and  other 
factors  will  guide  analysis  resolution. 


3. 2. 6. 2  Report  Analysis  Results 

The  presentation  of  analytical  results  may  be  as  important  as  the  analysis  itself.  The  results  must  be 
framed  in  terms  that  are  easily  understood  by  the  executive.  If  they  do  not  understand,  they  will  not 
use  the  analysis  in  making  the  decision.  It  is  also  important  to  capture  the  essential  elements  of  the 
analysis  to  facilitate  review  and  reuse  of  methods,  data,  and  algorithms. 

Often  an  analysis  will  age  for  a  considerable  period  of  time.  It  is  then  resurrected.  Good  documenta¬ 
tion  and  archiving  will  assure  that  the  analysis  can  be  explained  and  extended.  The  following  should 
be  practiced  routinely  during  and  at  the  conclusion  of  every  analysis: 

•  Maintain  consistency  between  analysis  results  and  issues  (iterative  process). 

•  Capture  and  document  lessons  learned. 

•  Document  analysis  details. 

•  Archive  analysis  methods,  data,  and  results. 
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4.  Analytic  Management  Best  Practices 


Executives  need  not  be  expert  in  quantitative  methods,  but  they  cannot  leave  the  field  to  experts. 

They  must  know  when  quantitative  methods  are  valuable  and  when  they  are  reliable.  They  must  have 
the  confidence  to  accept  or  override  the  recommendations  of  the  operations  research  staff  as  the 
situation  demands  and  must  appreciate  the  nature  and  role  of  corporate  systems  analysis.  The  man¬ 
ager  needs  to  understand  how  to  structure  a  large  or  complex  task  so  that  it  can  be  solved.15 


4.1  Management  Environment 

Managers  help  create  an  organizational  environment  where  analytical  excellence  flourishes.  This 
environment  includes  using  practices  that  fall  into  three  general  categories: 


4.1.1  Research 

Management  supports  research  to  understand  the  cognitive  processes  used  in  making  analysis  judg¬ 
ments,  but  needs  to  understand  how  thinking  skills  are  involved  in  analysis,  how  to  test  analysts  for 
these  skills,  how  to  train  analysts  to  improve  these  skills,  and  how  cognitive  limitations  affect  analy¬ 
sis  and  how  to  minimize  their  impact.  Simple  tools  and  techniques  help  prevent  avoidable  error. 


4.1.2  Training 

Managers  should  make  sure  that  analysts’  training  is  focused  on  organizational  procedures,  writing 
style,  and  methodological  techniques.  The  training  should  be  devoted  to  thinking  and  reasoning 
processes  involved  in  making  judgments,  and  to  trade  tools  available  to  alleviate  known  analysis  cog¬ 
nitive  problems. 


4.1.3  Exposure  to  Alternative  Mind-Sets 

Management  needs  to  ensure  that  well-reasoned  competing  views  have  the  opportunity  to  surface 
within  the  analysis  community.  Analysts  need  to  enjoy  a  sense  of  security  so  that  new  ideas  may  be 
expressed  with  minimal  fear  of  criticism  for  deviating  from  established  orthodoxy.  Management  can 
promote  activities  that  confront  analysts  with  alternative  perspectives,  e.g.,  consultation  with  outside 
experts,  analytical  debates,  competitive  analysis,  devil’s  advocates,  gaming,  and  interdisciplinary 
brainstorming.  Analysts  from  other  areas  who  are  not  specialists  in  the  subject  matter  should  also  be 
used. 


4.2  Analysis  Capability  Improvement  Management 

Management  should  mandate  a  systematic  analytical  process.  The  following  steps  define  the  process 
used  to  develop,  manage,  and  control  analysis  capability  improvements: 
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Identify  and  understand  current  and  future  analyses  problems. 

Define  the  objectives  of  expected  analyses 

-  To  understand  how  analysis  processes  are  performed 

-  To  improve  the  analysis  process — both  technical  and  managerial 
Collect  and  assess  current  analysis  capability  data  (establish  baseline) 

-  Determine  what  analyses  the  organization  is  doing. 

-  Collect  and  review  current  analysis  capability  data  (characterize  job). 

-  Identify  and  assess  current  organization  analysis  processes/tools. 

-  Interview  analysts  and  stakeholders  to  determine  processes  usage. 

Assess  future  organization  analysis  capabilities  needs 

-  Determine  what  analyses  the  organization  should  be  doing. 

-  Interview  analysts,  users,  and  stakeholders  to  identify  future  needs. 

-  Prepare  a  periodic  forecast  concerning  likely  areas  for  analysis  to  develop  resource 
plans,  analysis  capability  development  analysis,  and  background  materials  and  tech¬ 
nologies  to  facilitate  rapid  response  when  a  specific  issue,  topic,  or  analysis  question 
is  posed. 

Compare  baseline  capabilities  and  future  analysis  needs 

-  Use  the  SEI  CMMI  concepts  (or  ACMM)  as  a  model  for  processes/goals/practices. 

-  Identify  and  assess  areas  where  analysis  shortfalls  exist. 

-  Provide  recommendations  to  fill  the  gaps  (tools,  processes,  training,  etc.). 

Develop  and  implement  plans  to  update  organization  analysis  capabilities  as  required 

-  Establish  selection  criteria  (cost,  schedule,  performance). 

-  Prioritize  and  perform  make  or  buy  decisions.14 
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5.  Analysis  Capability  Maturity  Model  (ACMM) 


The  ACMM  is  a  process  framework  with  a  number  of  elements:  Specific  Goals,  Specific  Practices, 
Generic  Goals,  Generic  Practices,  and  maturity  levels.  Each  Specific  Goal  has  one  or  more  Specific 
Practices.  For  the  ACMM,  the  Specific  Goals  are  aggregated  into  four  process  areas:  Analysis 
Implementation,  Analysis  Management,  Analytic  Process  Management,  and  Analysis  Sustainment. 
These  elements  are  represented  in  Figure  5.1. 

Associated  with  each  Specific  Practice  are  Generic  Goals  appropriate  to  the  maturity  level.  The 
Generic  Goals  are  general  planning,  control,  implementation,  and  sustainment  goals  for  the  Specific 
Practices.  Generic  Goals  are  common  for  all  Specific  Practices  at  the  same  maturity  level.  Each 
Generic  Goal  has  one  or  more  Generic  Practices. 


ACMM:  The  foundation  for  appraising  and  improving 
analysis  capability  using  a  proven  best  practices  approach 


Analysis 

Appraisal 

Instrument* 


*  Based  on  the  Software  Engineering  Institute 's 
Capability  Maturin’  Model  Integrated  (CMMI) 
concepts  -  SE/SW/iPPD/SS,  Vl.l 


Analysis  Implementation 
Process  Areas 

Analysis  Management 
Process  Areas 

Analytic  Process 
Management 
Process  Areas 

Analysis 
Sustainment 
Process  Areas 

_ 

1 

Specific  Coals 


Specific  goals  unique  to  the 
process  area  that  describes 
what  must  be  implemented 
to  satisfy  the  processes. 


Common!  Features 

T 


f  Specific  1 

I  Specific  \ 

I  Specific  1 

l  Practices  J 

l  Practices  J 

A  Practices  J 

Commitment 

Ability 

Directing 

To  Perform 

To  Perform 

Implementation 

General  planning,  control, 
implementation,  and 
sustainment  goals  which 
apply  to  every  process. 


Verifying 

Implementation 


Creating  policies 
&  securing  sponsorship 


Ensuring 
implementation 
&  compliance 


Activities  considered  important  in  achieving 
specific  goals. 

Major  building  blocks  in  establishing 
organizational  analysis  capabilities. 


Establishing  and  maintaining 
plans,  resources,  assigned 
responsibility  &  authority,  and 
training 


Managing  processes 
performance,  product 
integrity,  &  stakeholder 
involvement 


Activities  that  ensure  processes  associated  with  analysis 
capabilities  will  be  effective,  repeatable,  and  lasting. 


Figure  5.1.  ACMM  overview. 
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5.1  Maturity  Levels 

The  ACMM  identifies  five  maturity  levels  that  correspond  to  the  CMMI  levels  defined  as  follows: 

1 .  Ad  hoc  processes  characterize  the  initial  level,  which  can  be  chaotic.  The  organization 
usually  does  not  provide  a  stable  environment.  Successes  in  these  organizations  depend 
on  the  competence  and  heroics  of  the  personnel  in  the  organization  and  not  on  the  use  of 
proven  processes.  Chaotic  organizations  can  produce  quality  studies,  but  they  frequently 
exceed  budget  and  schedule. 

2.  Managed  analysis  processes,  work  products,  and  services  for  individual  studies  charac¬ 
terize  the  managed  level.  The  organization  has  achieved  all  the  specific  and  generic 
goals  of  the  level-two  process  areas.  This  means  that  a  study  is  planned,  and  manage¬ 
ment  has  insight  into  the  status  of  work  products  and  services  at  defined  points  during 
execution.  However,  there  are  insufficient  processes  to  qualify  the  models  and  simula¬ 
tions,  data,  and  experts  used  in  the  study.  The  organization  is  very  focused  only  on  the 
work  at  hand  and  getting  the  job  out  the  door.  Estimation  of  schedule  and  resources  is  ad 
hoc  with  a  risk  to  study  success. 

3.  Standard  analysis  processes,  work  products,  and  services  for  the  analysis  organization 
characterize  the  defined  level.  The  organization  has  achieved  all  the  specific  and  generic 
goals  of  the  level-two  and  -three  process  areas.  This  means  that  there  is  a  greater  focus 
on  the  quality  of  the  inputs  to  an  analysis,  on  the  work  products  created,  and  on  the  final 
study  report.  Risk  is  managed.  Standard  organizational  processes  are  available  and  used 
to  perform  each  study.  The  organization  plans  for  the  capabilities  for  the  current  study. 

The  organization  also  forecasts  future  studies  and  plans  to  acquire  the  capabilities  to  per¬ 
form  them. 

4.  The  quantitative  level  is  characterized  by  analysis  tasks  being  managed  quantitatively. 

The  organization  has  achieved  all  the  specific  goals  of  level-two,  -three,  and  -four  proc¬ 
ess  areas  and  the  generic  goals  of  levels  two  and  three.  Quantitative  objectives  for  qual¬ 
ity  and  process  performance  are  established  and  used  as  criteria  in  managing  processes. 
Quality  and  process  performance  are  understood  in  statistical  terms  and  are  managed 
throughout  the  life  of  the  processes.  Quality  and  process  performance  measures  arc 
incorporated  into  the  organization’s  measurement  repository  to  support  fact-based 
decisions. 

5.  Continuous  improvements  of  analytic  processes  characterize  the  optimizing  level.  The 
organization  has  achieved  all  the  specific  goals  of  level-two,  -three,  -four,  and  -five  proc¬ 
ess  areas  and  the  generic  goals  of  levels  two  and  three.  Processes  are  improved  based  on 
the  quantitative  understanding  of  the  common  causes  of  variation.  The  focus  of  a  level- 
five  organization  is  on  incremental  and  innovative  improvements. 

In  the  following  sections,  the  standard  CMMI  maturity  levels  2  through  5  of  each  process  is  listed  in 
parenthesis  following  the  title  of  the  process. 
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5.2  Specific  Goals  and  Practices 

The  Specific  Goals  (SG)  and  Specific  Practices  (SP)  are  aggregated  into  four  process  areas:  Analysis 
Implementation,  Analysis  Management,  Analytic  Process  Management,  and  Analysis  Sustainment. 
This  aggregation  is  somewhat  arbitrary  and  is  used  as  a  convenient  taxonomy. 


5.2.1  Analysis  Implementation  Process  Area 

The  analysis  implementation  process  area  consists  of  related  processes  performed  to  achieve  an 
appropriate,  repeatable,  reliable,  and  objective  quantitative  analysis.  Six  of  these  process  areas  focus 
on  the  processes  used  to  actually  perform  the  analysis:  analysis  formulation,  analysis  and  data  plan¬ 
ning,  analysis  execution,  analysis  resolution,  and  analysis  reporting.  The  remaining  three  are  con¬ 
cerned  with  the  quality  of  the  inputs  to  the  analysis  and  include:  work  product  confirmation,  model 
and  simulation  corroboration,  and  qualification  of  subject  matter  experts. 

Analysis  Formulation  (2)  Analysis  Formulation  identifies  the  issue  to  be  analyzed  and  the  decision 
to  be  supported,  including  the  decision  maker.  The  time  frame  for  the  analysis  is  specified.  Analysis 
Formulation  also  identifies  stakeholders  to  the  decision  and  participants  in  the  analysis.  An  initial 
commitment  to  a  level  of  effort  to  be  expended  is  identified.  The  owner  of  the  issue,  often  the  deci¬ 
sion  maker,  who  may  be  assisted  by  an  analysis  organization,  conducts  Analysis  Formulation.  The 
Analysis  Formulation  product  is  frequently  called  the  Terms  of  Reference  (TOR)  for  the  analysis. 

SG  1  Manage  Analysis  Issues — Issues  are  identified  and  managed  and  inconsistencies  with  decisions 
and  analytic  products  are  identified. 

SP  1.1-1  Obtain  an  understanding  of  analysis  issues. 

SP  1 .2-1  Obtain  an  understanding  of  the  decisions  influenced  by  the  analysis. 

SP  1 .3-1  Develop  alternative  decisions  and  the  decision  criteria.* 

SP  1 .4-1  Obtain  commitment  to  the  issues  to  be  analyzed. 

SP  1.5-1  Manage  changes  in  analysis  issues. 

SP  1 .6-1  Maintain  bi-directional  traceability  of  issues  and  analysis  products. 

SP  1 .7-1  Identify  inconsistencies  between  analysis  products  and  issues. 

Analysis  and  Data  Planning  (2)  Analysis  and  Data  Planning  establishes  and  maintains  plans  that 
define  analysis  and  data  collection  activities  to  evaluate  possible  decisions  using  a  formal  evaluation 
that  compares  alternatives  using  quantified  criteria.  Once  the  analysis  question  is  understood,  study 
archives,  lessons  learned,  and  external  literature  are  surveyed  to  establish  an  analysis  baseline  for 
planning.  Efficient  planning  avoids  plowing  old  ground  and  builds  on  existing  bodies  of  knowledge. 
An  analysis  plan  designs  and  develops  the  algorithms  and  supporting  data  to  answer  the  analysis 
issues.  This  plan  is  established  and  maintained  as  the  basis  for  managing  the  analysis. 

SG  1  Establish  Estimates — Establish  and  maintain  analysis  planning  parameter  estimates. 


*  Adapted  from  CMMI,  Technical  Solution  to  identify  alternative  decisions 
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SP  1.1-1  Determine  an  analysis  baseline  founded  on  study  archives,  lessons  learned,  and 
external  literature. 

SP  1 .2-1  Estimate  the  scope  of  the  analysis. 

SP  1 .3-1  Establish  estimates  of  work  products  and  task  attributes. 

SP  1.4-1  Define  analysis  life  cycle. 

SP  1 .5-1  Determine  effort  and  cost  estimates. 


SG  2  Develop  an  Analysis  Plan- 
the  analysis. 


-Establish  and  maintain  an  analysis  plan  as  the  basis  for  managing 


SP  2.1-1 
SP  2.2-1 
SP  2.3-1 
SP  2.4-1 
SP  2.5-1 
SP  2.6-1 
SP  2.7-1 
SP  2.8-1 
SP  2.9-1 
SP  2.10-1 
SP  2.11-1 
SP  2.12-1 
SP  2.13-1 


Establish  guidelines  for  analysis. 

Establish  evaluation  criteria.1 
Select  Measures  of  Merit.1 

Identify  and  develop  alternative  analysis  approaches  and  selection  criteria.5 
Evaluate  alternative  approaches  to  the  analysis.* ** 

Select  analysis  approach.+t 

Develop  detailed  options  for  decision  and  selection  criteria.11 

Evolve  operational  concepts  and  scenarios  (context  in  which  options  are  evaluated).55 
Identify  analysis  risks. 

Plan  for  analysis  resources. 

Establish  the  analysis  plan. 

Determine  data  needs. 

Establish  a  technical  data  package. 111 


SG  3  Develop  a  Data  Plan — Establish  and  maintain  a  data  plan  as  the  basis  for  managing  the  data 
called  for  in  the  analysis  plan. 

SP  3.1-1  Plan  for  data  management. 

SP  3.2-1  Plan  for  data  certification. 

SP  3.3-1  Establish  data  descriptions.111 

SP  3.4-1  Analyze  potential  data  sources.555 

SP  3.5-1  Evaluate  and  determine  data  sources. 

SP  3.6-1  Perform  make,  acquire,  or  reuse  data  analyses. 

SP  3.7-1  Establish  the  data  plan. 


*  Source  CMM1,  DAR 

*  Source  CMMI,  DAR 
1  Source  CMMI,  DAR 

s  Source  CMMI,  Technical  Solution 
"  Source  CMMI,  DAR 
n  Source  CMMI,  DAR 

**  Source  CMMI,  Technical  Solution 
Source  CMMI,  Technical  Solution 
"  Source  CMMI,  Technical  Solution 
ttf  Source  CMMI,  Technical  Solution 
!J!  Source  CMMI,  Technical  Solution 
W  M&S,  real  world  data,  elicitation 
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SG  4  Integrate  the  Plans. 


SP  4.1-1  Integrate  the  analysis  and  data  plans.1 
SP  4.2-1  Plan  for  needed  knowledge  and  skills. 

SP  4.3-1  Plan  stakeholder  involvement. 

SP  4.4-1  Develop  analysis  and  data  support  documentation.* * 

SP  4.5-1  Establish  the  budget  and  schedule. 

SG  5  Obtain  Commitment  to  the  Plan  — Establish  and  maintain  analysis  plan  commitments. 

SP  5.1-1  Review  plans  that  affect  the  analysis. 

SP  5.2-1  Reconcile  work  and  resource  levels. 

SP  5.3-1  Obtain  plan  commitment . 

Analysis  Execution  (2)  Analysis  Execution  gathers  the  data,  performs  the  analysis  using  the  data, 
and  ensures  that  analytic  products  are  accurate  and  correct. 


SG  1  Prepare  for  Analysis  Execution. 

SP  1.1-1  Review  plans  for  completeness. 

SP  1 .2-1  Establish  the  analysis  environment. 

SP  1 .3-1  Determine  data  gathering  or  generation  sequence. 

SP  1 .4-1  Establish  data  formats  and  file  structures. 

SG  2  Data  Gathering  and  Certification. 

SP  2. 1 . 1  Collect  the  data  using  the  data  plan. 

SP  2.2- 1  Monitor  selected  data  gathering  processes. 

SP  2.3-1  Evaluate  selected  data  . 

SP  2.4-1  Revise  the  data  supplier  agreement  or  relationships. 

SP  2.5-1  Certify  data. 

SG  3  Analyze  the  Data  Product  Measures  of  Merit. 

SP  3. 1  -1  Confirm  readiness  of  data  components  for  aggregation. 
SP  3.2-1  Aggregate  data  into  Measures  of  Merit. 

SP  3.3-1  Package  analytic  work  products. 


Analysis  Reporting  (2)  Analysis  Reporting  reports  the  analytic  results,  documents  the  analytic  work 
products,  and  archives  the  analytic  products. 


*  Source  CMMI,  Technical  Solution 
1  Source  CMMI,  Technical  Solution 

*  Source  CMMI,  Technical  Solution 
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SG  1  Document  analysis  work  products. 


SP  1.1-1  Document  analysis  details. 

SP  1.2-1  List  incommensurables  and  other  factors. 

SG  2  Analysis  Reporting. 

SP  2 . 1  - 1  Report  final  analysis  interpretation. 

SG  3  Analysis  Archiving. 

SP  3.1-1  Archive  analysis  methods,  data,  results,  and  lessons  learned.* 

Analysis  Resolution  (3)  Analysis  Resolution  gives  meaning  to  the  analytic  work  products,  relates 
the  analysis  back  to  the  business  or  operation  problem,  and  identifies  factors  outside  the  scope  of  the 
analysis. 

SG  1  Interpret  Analysis  Work  Products. 

SP  1.1-1  Relate  analysis  to  the  issues  and  questions. 

SP  1 .2-1  Relate  measures  to  the  decision  criteria. 

SG  2  Identify  significant  factors  outside  the  scope  of  the  analysis. 

SP  2.1-1  List  incommensurables  and  other  factors. 

Work  Product  Confirmation  (3)  Work  Product  Confirmation  ensures  that  selected  work  products1 
meet  their  specified  needs. 

SG  1  Prepare  for  Confirmation. 

SP  1.1-1  Select  work  products  for  confirmation. 

SP  1.2-1  Establish  the  confirmation  environment. 

SP  1.3-1  Establish  confirmation  procedures  and  criteria. 

SG  2  Perform  Peer  Reviews. 

SP  2.1-1  Prepare  for  peer  reviews. 

SP  2.2-1  Conduct  peer  reviews. 

SP  2.3-1  Analyze  peer  review  data. 


*  Good  practice  to  have  a  taxonomy,  data  model,  for  analysis  to  facilitate  search. 
+  Applies  to  work  products  and  M&S. 


SG  3  Confirm  Selected  Work  Products. 


SP  3 . 1  - 1  Perform  confirmation. 

SP  3.2-1  Analyze  confirmation  results  and  identify  corrective  actions. 

Model  or  Simulation  Corroboration  (3)  Corroboration  demonstrates  that  a  model  or  simulation 
corresponds  to  the  real  world  with  sufficient  fidelity  for  the  intended  analyses.  Models  and  simula¬ 
tions  are  used  for  analysis  when  it  may  not  be  feasible  or  affordable  to  perform  statistically  significant 
experiments  in  the  real  world.  However,  there  are  usually  portions  of  the  model  that  can  be  compared 
to  real-world  data.  Corroboration  selects  those  elements  of  the  model  or  simulation  for  comparison  to 
the  real-world  data.  Conclusions  from  corroboration  are  used  to  support  models  and  simulations 
accreditation. 

SG  1  Prepare  for  Corroboration — Preparation  for  corroboration  is  conducted. 

SP  1.1-1  Select  model  or  simulation  for  corroboration. 

SP  1 .2-1  Select  features  of  the  model  or  simulation  for  corroboration. 

SP  1 .3-1  Identify  the  real-world  data  to  be  used  for  comparison. 

SP  1 .4-1  Establish  the  corroboration  environment. 

SP  1 .5-1  Establish  corroboration  procedures  and  criteria. 

SG  2  Corroborate  the  Model  or  Simulation — The  models  or  simulations  are  corroborated  to  ensure 
that  they  are  true  to  the  real  world. 

SP  2.1-1  Generate  synthetic  data  for  corroboration  using  models  or  simulations. 

SP  2.2-1  Compare  synthetic  data  to  the  real-world  data. 

SP  2.3-1  Analyze  the  agreement  between  the  synthetic  and  real-world  datasets. 

Subject  Matter  Expert  Qualification  (3)  Subject  Matter  Expert  (SME)  Qualification  selects  indi¬ 
viduals  with  the  requisite  knowledge  areas  to  opine  on  selected  issues  of  an  analysis.  SMEs  are  often 
used  to  create  importance  rankings  for  decision  criteria. 

SG  1  Develop  Qualifications — Needed  qualifications  for  SME(s)  are  developed. 

SP  1.1-1  Select  analysis  issue  for  the  SMEs. 

SP  1 .2-1  Identify  knowledge  or  expertise  needed  for  the  analysis  issue. 

SP  1 .3-1  Establish  desired  credentials  for  the  SME. 

SG  2  Select  the  SME — The  desired  credentials  are  compared  to  individual  candidates. 

SP  2 . 1  - 1  Generate  prospective  SME  pool . 

SP  2.2-1  Elicit  SME  candidates’  credentials. 

SP  2.3-1  Compare  SME  candidates’  credentials  with  the  desired  credentials. 

SP  2.4-1  Select  SME  from  the  candidate  pool. 


19 


5.2.2  Analysis  Management  Process  Areas 

The  analysis  management  process  area  consists  of  related  processes  performed  to  manage  analyses. 
One  specific  process  deals  with  understanding  progress  against  the  plan.  Another  deals  with  risk 
management.  The  use  of  defined  processes,  organization,  and  operation  of  the  internal  study  team  are 
the  subjects  of  a  process  area.  Two  specific  processes — analysis  provider  agreement  and  manage¬ 
ment  of  external  stakeholder  involvement — deal  with  management  external  to  the  organization.  This 
process  area  contains  a  level-four  process  area:  quantitative  analysis  management. 

Analysis  Monitoring  and  Control  (2)  Analysis  Monitoring  and  Control  provides  an  understanding 
of  the  analysis’s  progress  so  that  appropriate  corrective  actions  can  be  taken  when  the  analysis’s  per¬ 
formance  deviates  significantly  from  the  plan. 

SG  1  Monitor  Analysis  Against  Plan — Actual  performance  and  progress  of  the  analysis  are  monitored 


against  the  analysis  plan. 

SP  1.1-1 

Monitor  analysis  planning  parameters. 

SP  1.2-1 

Monitor  commitments. 

SP  1.3-1 

Monitor  analysis  risks. 

SP  1.4-1 

Monitor  data  management. 

SP  1.5-1 

Monitor  stakeholder  involvement. 

SP  1.6-1 

Conduct  progress  reviews. 

SP  1.7-1 

Conduct  milestone  reviews. 

SG  2  Manage  Corrective  Action  to  Closure — Corrective  action  is  managed  to  closure  when  the  analy¬ 
sis’s  performance  or  results  deviate  significantly  from  the  plan. 

SP  2. 1  -1  Analyze  issues. 

SP  2.2-1  Take  corrective  action. 

SP  2.3-1  Manage  corrective  action. 

Analysis  Provider  Agreement  (2)  Analysis  Provider  Agreement  manages  analyses  received  from 
providers,  for  which  there  exists  a  formal  agreement. 

SG  1  Establish  Provider  Agreements — Agreements  with  the  providers  are  established  and  maintained. 

SP  1.1-1  Specify  analysis  to  be  provided. 

SP  1 .2-1  Determine  potential  providers. 

SP  1.3-1  Select  providers. 

SP  1 .4-1  Establish  provider  agreements. 

SG  2  Satisfy  Provider  Agreements — Agreements  with  the  providers  are  satisfied  by  both  the  analysis 
and  the  provider. 

SP  2.1-1  Review  products. 
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SP  2.2-1 
SP  2.3-1 
SP  2.4-1 


Execute  the  provider  agreement. 
Accept  the  provider  product. 
Transition  products. 


Management  of  External  Stakeholder  Involvement  (3)  Management  of  External  Stakeholder 
involvement  establishes  and  manages  the  analysis  and  the  relevant  stakeholders  involvement  accord¬ 
ing  to  an  integrated  and  defined  process  that  is  tailored  from  the  organization’s  set  of  standard 
processes. 

SG  1  Coordinate  and  Collaborate  with  Relevant  Stakeholders. 

SP  1.1-1  Manage  stakeholder  involvement. 

SP  1.2-1  Manage  dependencies. 

SP  1.3-1  Resolve  coordination  issues. 

SG  2  Develop  the  shared  vision  for  External  Stakeholder  Involvement  management. 

SP  2.1-1  Define  analysis’s  shared-vision  context. 

SP  2.2-1  Establish  the  analysis’s  shared  vision. 

SG  3  Organize  Integrated  Teams  for  Analysis. 

SP  3.1-1  Determine  the  integrated  team  structure  for  the  analysis. 

SP  3.2-1  Develop  a  preliminary  distribution  of  analysis  activities  to  external  stakeholders. 

SP  3.3.1  Identify  external  stakeholder  data  sources. 

SP  3.4-1  Establish  integrated  teams. 

Risk  Management  (3)  Risk  Management  identifies  potential  problems  before  they  occur  so  that  risk¬ 
handling  activities  may  be  planned  and  invoked  as  needed  across  the  life  of  the  product  or  analysis  to 
mitigate  adverse  impacts  on  achieving  objectives. 

SG  1  Prepare  for  Risk  Management  [PA148.IG101]. 

SP  1.1-1  Determine  risk  sources  and  categories. 

SP  1 .2-1  Define  risk  parameters. 

SP  1 .3-1  Establish  a  risk  management  strategy. 

SG  2  Identify  and  Analyze  Risks. 

SP  2.1-1  Identify  risks. 

SP  2.2-1  Evaluate,  categorize,  and  prioritize  risks. 

SG  3  Mitigate  Risks. 
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SP  3.1-1  Develop  risk  mitigation  plans. 
SP  3.2-1  Implement  risk  mitigation  plans. 


Management  of  the  Internal  Analysis  Team  (3)  The  Internal  Analysis  Team  management  forms 
and  sustains  an  internal  analysis  team  for  analysis  products  development.  Internal  analysis  team 
management  also  covers  a  shared-vision  establishment  for  the  analysis  and  a  team  structure  for  inte¬ 
grated  teams  that  will  carry  out  the  analysis  objectives. 


SG  1  Use  the  Analysis  Defined  Process. 


SP  1.1-1 
SP  1.2-1 
SP  1.3-1 
SP  1.4-1 
SP  1.5-1 


Establish  the  defined  process  for  the  analysis  effort. 

Use  organizational  process  assets  for  planning  analysis  activities. 
Integrate  plans. 

Manage  the  analysis  using  the  integrated  plans. 

Contribute  to  the  organizational  process  assets. 


SG  2  Establish  Team  Composition. 


SP  2.1-1  Identify  team  tasks. 

SP  2.2-1  Identify  needed  knowledge  and  skills. 
SP  2.3-1  Assign  appropriate  team  members. 

SG  3  Govern  Team  Operation. 

SP  3.1-1  Establish  a  shared  vision. 

SP  3.2-1  Establish  a  team  charter. 

SP  3.3-1  Define  roles  and  responsibilities. 

SP  3.4-1  Establish  operating  procedures. 

SP  3 . 5  - 1  Coll  aborate  among  interfacing  teams . 


Quantitative  Analysis  Management  (4)  The  Quantitative  Analysis  Management  process  area  quan¬ 
titatively  manages  the  Analysis’s  defined  process  to  achieve  the  Analysis’s  established  quality  and 
process-performance  objectives. 


SG  1  Quantitatively  Manage  the  Analysis. 

SP  1 . 1  -1  Establish  the  analysis’s  objectives. 

SP  1 .2-1  Compose  the  defined  process. 

SP  1 .3-1  Select  the  sub-processes  that  will  be  statistically  managed. 
SP  1 .4-1  Manage  analysis  performance. 

SG  2  Statistically  Manage  Sub  process  Performance. 

SP  2.1-1  Select  measures  and  analytic  techniques. 
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SP  2.2-1  Apply  statistical  methods  to  understand  variation. 

SP  2.3-1  Monitor  performance  of  the  selected  sub-processes. 

SP  2.4-1  Record  statistical  management  data. 

5.2.3  Analytic  Process  Management  Process  Area 

The  Analytic  Process  Management  Process  Area  consists  of  related  processes  performed  to  achieve  a 
set  of  process  management  goals.  The  five  process  areas  within  this  category  are  for  the  organization 
and  are  level  three  or  higher.  All  of  these — Organizational  Process  Focus,  Organizational  Process 
Definition,  Organizational  Training,  Organizational  Process  Improvement,  and  Organizational  Inno¬ 
vation  and  Deployment — focus  on  establishing,  maintaining,  and  improving  organizational  processes. 

Organizational  Process  Focus  (3)  The  Organizational  Process  Focus  plans  and  implements  organi¬ 
zational  process  improvement  based  on  a  thorough  understanding  of  the  current  strengths  and  weak¬ 
nesses  of  the  organization’s  processes  and  process  assets. 

SG  1  Determine  Process-Improvement  Opportunities. 

SP  1.1-1  Establish  organizational  process  needs. 

SP  1.2-1  Appraise  the  organization’s  processes. 

SP  1.3-1  Identify  the  organization’s  process  improvements. 

SG  2  Plan  and  Implement  Process-Improvement  Activities. 

SP  2.1-1  Establish  process  action  plans. 

SP  2.2-1  Implement  process  action  plans. 

SP  2.3-1  Deploy  organizational  process  assets. 

SP  2.4-1  Incorporate  process-related  experiences  into  the  organizational  process  assets. 

Organizational  Process  Definition  (3)  Organizational  Process  Definition  establishes  and  maintains 
a  usable  set  of  organizational  process  assets. 

SG  1  Establish  Organizational  Process  Assets. 

SP  1.1-1  Establish  standard  processes. 

SP  1.2-1  Establish  life-cycle  model  descriptions. 

SP  1.3-1  Establish  tailoring  criteria  and  guidelines. 

SP  1 .4-1  Establish  the  organization’s  measurement  repository. 

SP  1 .5-1  Establish  the  organization’s  process  asset  library. 

Organizational  Training  (3)  Organizational  Training  develops  the  skills  and  knowledge  of  people 
so  they  can  perform  their  roles  effectively  and  efficiently. 

SG  1  Establish  an  Organizational  Training  Capability. 
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SP  1.1-1  Establish  the  strategic  training  needs. 

SP  1 .2-1  Determine  which  training  needs  are  the  organization’s  responsibility. 

SP  1 .3-1  Establish  an  organizational  training  tactical  plan. 

SP  1.4-1  Establish  training  capability. 

SG  2  Provide  Necessary  Training. 

SP  2. 1  - 1  Deliver  training. 

SP  2.2-1  Establish  training  records. 

SP  2.3-1  Assess  training  effectiveness. 

Organizational  Process  Performance  (4)  Organizational  Process  Performance  establishes  and 
maintains  a  quantitative  understanding  of  the  performance  of  the  organization’s  set  of  standard  proc¬ 
esses  in  support  of  quality  and  process-performance  objectives,  and  provides  the  process  performance 
data,  baselines,  and  models  to  quantitatively  manage  the  organization’s  analyses. 

SG  1  Establish  Performance  Baselines  and  Models. 

SP  1 . 1  - 1  Select  processes. 

SP  1 .2-1  Establish  process  performance  measures. 

SP  1 .3-1  Establish  quality  and  process-performance  objectives. 

SP  1 .4-1  Establish  process  performance  baselines. 

SP  1.5-1  Establish  process  performance  models. 

Organizational  Innovation  and  Deployment  (5)  The  Organizational  Innovation  and  Deployment 
process  area  selects  and  deploys  incremental  and  innovative  improvements  that  measurably  improve 
the  organization’s  processes  and  technologies.  The  improvements  support  the  organization’s  quality 
and  process-performance  objectives  as  derived  from  the  organization’s  business  objectives. 

SG  1  Select  Improvements. 

SP  1.1-1  Collect  and  analyze  improvement  proposals. 

SP  1 .2-1  Identify  and  analyze  innovations. 

SP  1.3-1  Pilot  improvements. 

SP  1 .4- 1  Select  improvements  for  deployment. 

SG  2  Deploy  Improvements. 

SP  2 . 1  - 1  Plan  the  deployment. 

SP  2.2-1  Manage  the  deployment. 

SP  2.3-1  Measure  improvement  effects. 
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5.2.4  Analysis  Sustainment  Process  Area 

The  Analysis  Sustainment  Process  Area  contains  processes  that  are  used  by  processes  in  other  areas. 
These  processes  are  performed  to  sustain  analysis  capability,  measure  analytic  performance,  and  pro¬ 
vide  quality  analysis.  One  of  the  most  significant  processes  areas  for  the  ACMM  is  Future  Analysis 
Capabilities  Development.  It  may  not  be  feasible  for  an  analysis  organization  to  develop  new  capa¬ 
bility  when  it  is  called  upon  to  develop  quantitative  information;  the  decision  horizon  is  often  close. 
Consequently,  the  effective  analysis  organization  must  be  proactive  and  seek  to  have  the  needed  skills 
and  tools  at  hand.11 

Measurement  and  Analysis  (2)  Measurement  and  Analysis  develops  and  sustains  a  measurement 
capability  that  is  used  to  support  management  information  needs. 

SG  1  Align  Measurement  and  Analysis  Activities — Measurement  objectives  and  activities  are  aligned 
with  identified  information  needs  and  objectives. 

SP  1.1-1  Establish  measurement  objectives. 

SP  1 .2-1  Specify  measures. 

SP  1 .3-1  Specify  data  collection  and  storage  procedures. 

SP  1 .4-1  Specify  analysis  procedures. 

SG  2  Provide  Measurement  Results — Measurement  results  that  address  identified  information  needs 
and  objectives  are  provided. 

SP  2. 1-1  Collect  measurement  data. 

SP  2.2-1  Analyze  measurement  data. 

SP  2.3-1  Store  data  and  results. 

SP  2.4-1  Communicate  results. 

Analysis  Process  and  Analytic  Product  Quality  Assurance  (2)  Analysis  Process  and  Analytic 
Product  Quality  Assurance  provide  staff  and  management  with  objective  insight  into  processes  and 
associated  work  products. 

SG  1  Objectively  Evaluate  Processes  and  Work  Products — Adherence  of  the  performed  process  and 
associated  work  products  and  services  to  applicable  process  descriptions,  standards,  and  procedures  is 
objectively  evaluated. 

SP  1.1-1  Objectively  evaluate  analysis  processes. 

SP  1.2-1  Objectively  evaluate  analytic  work  products  and  analysis  services. 

SG  2  Provide  Objective  Insight — Noncompliance  issues  are  objectively  tracked  and  communicated, 
and  resolution  is  ensured. 

SP  2.1-1  Communicate  and  ensure  noncompliance  issues  resolution. 
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SP  2.2-1  Establish  records. 


SG  3  Analysis  Review 

SP  3.1-1  Perform  an  analysis  independent  review. 

Configuration  Management  (2)  Configuration  Management  establishes  and  maintains  the  integrity 
of  work  products  using  configuration  identification,  configuration  control,  configuration  status 
accounting,  and  configuration  audits. 

SG  1  Establish  Baselines — Identified  work  product  baselines  are  established. 

SP  1.1-1  Identify  configuration  items. 

SP  1 .2-1  Establish  a  configuration  management  system. 

SP  1 .3-1  Create  or  release  baselines. 

SG  2  Track  and  Control  Changes — Changes  to  the  work  products  under  configuration  management 
are  tracked  and  controlled. 

SP  2 . 1  - 1  Track  change  requests. 

SP  2.2-1  Control  configuration  items. 

SG  3  Establish  Integrity— Baseline  integrity  is  established  and  maintained. 

SP  3.1-1  Establish  configuration  management  records. 

SP  3.2-1  Perform  configuration  audits. 

Establish  and  Maintain  Analysis  Capabilities  (3)  The  purpose  of  Establish  and  Maintain  Analyses 
Capabilities  is  to  prepare  for  current  analysis  activities. 

SG  1  Acquire  Capabilities. 

SP  1 . 1  - 1  Accomplish  changes  made  necessary  by  the  study 

SP  1.2-1  Perform  construct,  acquire,  or  modify  existing  capabilities  trades. 

SP  1 .3-1  Construct  or  acquire  capabilities. 

SG  2  Manage  and  develop  analysts. 

SP  2.1-1  Identify  unique  skill  requirements. 

SP  2.2-1  Establish  leadership  mechanisms. 

SP  2.3-1  Establish  incentives  for  development. 


’  Apply  software  acquisition  CMM,  perhaps. 
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SP  2.4-1 


Establish  mechanisms  to  balance  team  and  home  organization  responsibilities. 


SG  3  Establish  and  maintain  mechanisms  for  analysis. 

SP  3. 1  -1  Identify  modeling,  simulation,  and  data  processing  needs. 

SP  3.2-1  Confirm  analysis  components  readiness. 

SG  4  Establish  and  maintain  facilities  for  analysis. 

SP  4.1-1  Establish  the  organization’s  shared  vision. 

SP  4.2-1  Establish  an  integrated  work  environment. 

Future  Analysis  Capabilities  Development  (3)  The  purpose  of  Future  Analysis  Capabilities  Devel¬ 
opment  is  to  identify  future  analyses  topic  areas  and  to  develop  the  future  capabilities  needed  to  per¬ 
form  those  analyses. 

SG  1  Develop  Future  Analysis  Topic  Areas. 

SP  1.1-1  Collect  stakeholder  needs. 

SP  1 .2-1  Elicit  additional  stakeholder  needs. 

SP  1.3-1  Develop  the  customer  issues. 

SG  2  Develop  Future  Analysis  Concepts  and  Mechanisms. 

SP  2.1-1  Identify  candidate  future  analysis  concepts. 

SP  2.2-1  Identify  alternative  mechanisms  to  accomplish  future  analysis  concepts. 

SP  2.3-1  Select  concepts  and  mechanisms  to  perform  future  analyses. 

SG  3  Develop  Future  Capability  Requirements. 

SP  3 . 1  - 1  Establish  analysis  scenarios. 

SP  3.2-1  Establish  a  definition  of  required  functionality. 

SP  3.3-1  Develop  requirements. 

SP  3.4-1  Analyze  requirements. 

SP  3.5-1  Estimate  future  capability  schedule  and  resources. 

SP  3.6-1  Budget  for  future  capabilities. 

SG  4  Develop  Future  Analyses  Plans. 

SP  4.1-1  Plan  for  future  analyses.  The  organization  develops  resource  plans,  analysis 

capability  development  projects,  and  basic  background  materials  and  technolo¬ 
gies.  (This  facilitates  rapid  response  when  a  specific  issue  or  topic  analysis 
question  is  posed.) 

SP  4.2-1  Identify  future  decisions  for  quantitative  analysis. 
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SP  4.3-1  Identify  decision  takers. 

SP  4.4-1  Identify  time  analysis  is  needed. 

Causal  Analysis  and  Resolution  (5)  Causal  Analysis  and  Resolution  identifies  causes  of  defects  and 
other  problems  and  takes  action  to  prevent  them  from  occurring  in  the  future. 

SG  1  Determine  Causes  of  Defects. 

SP  1.1-1  Select  defect  data  for  analysis. 

SP  1.2-1  Analyze  causes. 

SG  2  Address  Causes  of  Defects. 

SP  2.1-1  Implement  the  action  proposals. 

SP  2.2-1  Evaluate  the  effect  of  changes. 

SP  2.3-1  Record  data. 

5.3  Generic  Goals  and  Practices 

The  generic  goals  and  practices  used  in  the  ACMM  are  unmodified  from  those  used  in  the  SEI  CMMI 
model  and  are  presented  here  only  for  document  completeness. 


5.3.1  Generic  Goals 

General  planning,  control,  implementation,  and  sustainment  goals  that  apply  to  every  process. 

5.3.2  Common  Features 

Commitment  To  Perform:  Creating  policies  and  securing  sponsorship 

Ability  To  Perform:  Establishing  and  maintaining  plans,  resources,  assigned  responsibility  &  author¬ 
ity,  and  training 

Directing  Implementation:  Managing  processes  performance,  product  integrity,  and  stakeholder 
involvement 

Verifying  Implementation:  Ensuring  implementation  and  compliance 


5.3.3  Generic  Practices 

Activities  that  ensure  processes  associated  with  analysis  capabilities  will  be  effective,  repeatable,  and 
lasting. 
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GG  1  Achieve  Specific  Goals — Transform  identifiable  input  work  products  to  produce  identifiable 
output  work  products. 

GP  1.1  Perform  Base  Practices  to  develop  work  products  and  provide  services  to  achieve  the 
specific  goals. 

GG  2  Institutionalize  a  Managed  Process — Process  is  institutionalized  as  a  managed  process. 


GP  2.1 
GP  2.2 
GP  2.3 

GP  2.4 

GP  2.5 
GP  2.6 
GP  2.7 
GP  2.8 

GP  2.9 

GP  2.10 


Establish  Organizational  Policy  for  planning  and  maintaining  the  process. 

Establish  and  maintain  the  plan  for  performing  the  process. 

Provide  adequate  resources  for  performing  the  process,  developing  the 
work  products,  and  providing  the  process  services. 

Assign  responsibility  and  authority  for  performing  the  process,  developing 
the  work  products,  and  providing  the  process  services. 

Train  the  people  performing  or  supporting  the  process  as  needed. 

Manage  designated  process  work  products  configuration. 

Identify  and  involve  the  relevant  stakeholders  as  planned. 

Monitor  and  control  the  process  against  plans  for  performing  the  process 
and  take  appropriate  corrective  action. 

Objectively  evaluate  adherence  of  process  against  its  process  description, 
standards,  and  procedures,  and  address  noncompliance. 

Review  process  activities,  status,  and  results  with  higher-level  management 
and  resolve  issues. 


GG  3  Institutionalize  a  Defined  Process — Process  is  institutionalized  as  a  defined  process. 

GP  3. 1  Establish  and  maintain  the  defined  process  description. 

GP  3.2  Collect  improvement  information  including  work  products,  measures,  meas¬ 
urement  results,  and  information  derived  from  planning  and  performing  the 
process  to  support  future  use  and  the  organization’s  processes  and  process 
assets  improvement. 

GG  4  Institutionalize  a  Quantitatively  Managed  Process — Process  is  institutionalized  as  a  quantita¬ 
tively  managed  process. 

GP  4. 1  Establish  and  maintain  quantitative  objectives  for  the  capability  that  address 
quality  and  process  performance  based  on  customer  needs. 
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GP  4.2  Stabilize  the  sub-process  performance  to  determine  the  ability  of  the  process 
to  achieve  the  established  quantitative  quality  and  process-performance 
objectives. 

GG  5  Institutionalize  an  Optimizing  Process — Process  is  institutionalized  as  an  optimizing  process. 

GP  5.1  Ensure  continuous  capability  improvement  in  filling  the  organization’s  objectives. 
GP  5.2  Identify  and  correct  root  causes  of  defects  and  other  capability  problems. 
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6.  Analysis  Capability  Appraisal 


Two  types  of  appraisals  can  be  conducted:  a  staged  appraisal  or  a  continuous  appraisal.  For  a  staged 
appraisal,  each  Specific  Process  and  its  associated  Generic  Practices  are  appraised  at  a  given  or 
greater  level  of  maturity  and  the  organization  is  appraised  at  the  lowest  maturity  level  attained  for  any 
Specific  Practice.  For  a  continuous  appraisal,  a  maturity  level  is  determined  for  each  Specific  Prac¬ 
tice  and  its  associated  Generic  Practice.1213 

An  appraisal  evaluates  the  analysis  organization’s  capability  for  producing  high-quality  analysis 
products  and  services.  Independent  appraisals  are  taken  seriously,  even  though  they  may  report  what 
the  program  staff  already  knows,  because  the  appraisal  team’s  messages  often  carry  greater  weight 
and  reach  higher  levels  in  the  organization.  When  a  sponsor  requests  an  appraisal,  the  time  and 
expense  will  likely  impact  the  participating  organizations.  Therefore,  the  team  should  plan  to  make 
efficient  use  of  the  participant’s  time  as  they  may  already  be  overworked,  and  the  appraisal  may  be 
seen  as  an  impediment  to  progress.  Reviewing  current  analysis  issues  in  the  organization  and  its 
projected  future  analysis  issues  indicates  analysis  support  needs. 

The  appraisal  initially  focuses  on  current  analysis  work  and  compares  this  work  with  the  analysis 
needed  to  support  the  organization’s  mission.  The  appraisal  examines  the  analysis  processes  that  are 
used  with  an  emphasis  on  initiation,  planning,  execution,  timeliness,  and  quality  assurance.  It  exam¬ 
ines  the  tools,  models,  and  simulations  currently  in  use  and  determines  their  status  with  regard  to  veri¬ 
fication  (the  tool  meets  its  design),  validation  (the  tool  adequately  represents  the  phenomena  under 
study),  and  accreditation  (there  exists  an  authoritative  statement  of  the  tool’s  suitability  to  analyze 
specific  issues).  Of  special  interest  for  analysis  tools  is  the  appropriateness  of  the  level  of  detail,  the 
ease  with  which  sensitivity  analysis  can  be  performed,  and  the  experts’  credentials  used  in  analysis 
studies.  The  appraisal  recommends  changes  to  the  processes  being  used,  identifies  tool  shortfalls,  and 
recommends  development  projects  to  close  any  analysis  deficiencies. 


6.1  Task  Objective 

To  develop  the  approach  and  the  methodology  to  be  used  by  an  appraisal  team  to  conduct  an  inde¬ 
pendent,  objective  assessment  of  an  organization’s  analysis  capabilities  utilizing  the  ACMM  model. 


6.2  Appraisal  Purpose 

The  purpose  of  performing  an  independent  appraisal  of  the  state  of  analysis  capability  in  an  analysis 
or  operations  research  organization  is  to  enhance  the  organization’s  analysis  processes.  Documenta¬ 
tion  of  both  current  analysis  capabilities  and  future  analysis  needs  to  support  the  mission  is  required. 
To  accomplish  this,  the  current  tools,  models,  and  simulations  must  be  examined  and  their  use  con¬ 
firmed  and  corroborated.  The  result  is  a  well-understood  foundation  (baseline)  for  a  continuous  proc¬ 
ess  improvement  to  satisfy  future  needs  with  recommendations  for  specific,  measurable 
improvements. 
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6.3  Appraisal  Approach 

Access  to  information  will  come  through  a  series  of  interviews  with  each  group  and  through  relevant 
documentation  examination.  The  interviews  will  be  used  to  identify  analysis  team  leaders  and  spon¬ 
sors;  identify  future  analysis  areas  and  topics;  understand  analysis  processes,  their  use,  and  manage¬ 
ment;  identify  any  process  documents  and  capability  descriptions;  and  identify  accreditation  docu¬ 
ments  along  with  the  supporting  verification  and  accreditation  documentation  for  computer  models 
and  simulations  in  use.  The  approach  includes: 

•  Examine  the  organization  and  evidential  documentation  of  current  analysis  processes, 
tools,  models  and  simulations,  and  related  activities. 

•  Conduct  interviews  with  selected  members  of  the  organization,  including  management,  to 
identify  both  current  and  future  analysis  needs. 

•  Identify  gaps  in  the  existence  and/or  use  of  current  processes,  tools,  and  models  and 
simulations,  and  future  needs  as  a  result  of  the  independent  team’s  appraisal  of  the  orga¬ 
nization’s  analysis  capabilities. 

•  Make  no  quality  judgments — only  the  processes  existence  and  usage  are  evaluated. 

•  Make  no  attributions  to  specific  organizations/programs  or  individual  interviewees  so  that 
interviewees  can  be  candid  and  forthcoming,  without  fear  of  reprisal. 

•  Results  and  recommendations  are  made  available  to  management  to  be  used,  as  they 
deem  appropriate. 

6.4  Analysis  Appraisal  and  Processes  Improvement 

Figure  6. 1  provides  an  overview  of  the  process  used  for  the  analysis  capability  appraisal  and  for 
analysis  product  and  analysis  process  improvement.  These  portray  the  high-level  activities  and  flow 
of  events  that  should  occur  when  performing  an  appraisal  and  how  these  processes  fit  into  process 
management  and  improvement  activities.  The  following  definitions  are  made  to  ensure  that  a  distinc¬ 
tion  is  made  between  the  two  processes. 

Definitions: 

Analysis  Process:  Process  used  by  the  organization  to  analyze  decision  questions  and  issues  for  the 
purpose  of  developing  a  supportable,  quantitative  basis  for  executive  decisions. 

Analysis  Management  Process:  Methods  by  which  the  organization  identifies  issues  and  acquires 
capabilities  (facilities,  hardware/software,  people  skills,  procedures,  and  data)  to  satisfy  both  current 
and  future  analysis  needs. 
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Figure  6. 1 .  Analysis  appraisal  and  process  improvement. 


6.5  Typical  Independent  Appraisal  Outcomes 

An  independent  appraisal  provides  an  objective  evaluation  of  an  organization’s  analysis  capabilities. 
This  provides  the  basis  for  understanding  both  the  current  (baseline)  capabilities  and  the  future  analy¬ 
sis  process  capability  needs.  Once  this  understanding  is  achieved,  a  plan  of  action  can  be  generated 
and  implemented  to  assure  the  organization  will  be  able  to  provide  quality  analyses  now  and  in  the 
future.  Specific  appraisal  outcomes  might  include: 

•  Common  understanding  of  the  analysis  role  in  the  organization 

•  Recommendations  for  monitoring,  controlling,  and  reporting  analyses  within  the  organi¬ 
zation  and  with  the  customer  and  relevant  stakeholders 

•  Appraisal  of  the  degree  of  analysis  collaboration  and  coordination  in  use 

•  Recommended  analysis  processes,  if  needed 

•  Suggested  products  and/or  process  improvement  opportunities 

•  Identification  of  potential  problems 

•  Recommendations  for  management  strategies 

•  Highlighting  of  successes  and  good  practices 
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6.6  Independent  Appraisal  Team  Role 

The  role  that  the  appraisal  team  plays  in  the  process  is  vital  to  the  success  of  the  appraisal.  Objectiv¬ 
ity  of  the  team’s  appraisal  is  the  only  way  to  assure  that  the  results  are  not  biased  or  influenced  by 
factors  within  the  organization. 


6.6.1  Team  Structure 

The  team  is  structured  to  be  independent  and  objective,  thus  providing  no  potential  conflict  of  inter¬ 
est.  Independent  thinking  can  yield  creative  solutions  and  will  be  perceived  as  unbiased  by  inter¬ 
viewees.  The  team  should  be  small  enough  to  allow  easy  coordination  and  yet  have  enough  members 
to  assure  coverage  if  some  team  members  are  not  always  available.  There  should  always  be  at  least 
two  team  members  present  during  interviews.  An  open  environment  for  information  exchange  allows 
leveraging  existing  staff  knowledge  and  insight  across  all  organization  levels.  The  team  members 
should  bring  a  diversity  of  backgrounds  (and  opinions)  that  will  help  in  achieving  balanced  decisions. 
Appraisal  team  members  should  be  selected  with  regard  to  independence;  that  is,  they  should  not  be 
past  or  current  members  of  the  organization  being  assessed  or  any  supporting  or  heritage  organiza¬ 
tions.  A  person  with  stature  in  the  operations  research/systems  analysis  community  with  the  appro¬ 
priate  clearances  should  be  selected  to  lead  this  effort.  Experience  conducting  CMM  appraisals  is 
highly  desirable. 


6.6.2  Appraisal  Team  Activities 

An  independent  appraisal  is  a  complex  and  labor-intensive  process.  Adequate  resources  must  be 
applied  for  the  appraisal  to  be  successful.  The  more  comprehensive  the  up-front  preparation  is,  the 
greater  the  chance  for  success  with  the  available  resources.  The  expertise  and  experience  of  the  team 
leader  and  the  dedication  of  the  team  members  is  vital  to  the  appraisal’s  success.  The  team  should  be 
provided  with  unfettered  access  to  key  people  and  data  to  assure  that  the  results  are  credible.  Some 
typical  appraisal  team  activities  include  the  following: 

•  Select  appraisal  team  leader.  This  is  the  responsibility  of  the  sponsor. 

•  Select  appraisal  team  members  including  group  facilitators.  Facilitators  serve  as  the  liai¬ 
son  to  respective  groups,  identify  interviewees,  facilitate  interviews,  pursue  and  compile 
detailed  group  information  (baseline  descriptions)  from  the  respective  groups,  and  pro¬ 
vide  in-depth  support  to  the  appraisal  team. 

•  Train  the  appraisal  team.  Training  materials  should  include  appraisal  planning,  analysis 
planning,  and  independent  appraisal  team  lessons  learned. 

•  Prepare,  approve,  and  use  the  appraisal  instrument  (ACMM)  as  the  appraisal  basis. 

•  Identify  internal  team  members  from  each  of  the  organization’s  groups  to  act  as  liaisons. 

•  Meet  with  group  facilitators  to  identify  interview  candidates. 

•  Participate  with  and  support  internal  group  interviews,  and  document  review  activities. 

•  Conduct  organization  management  interviews;  capture  concerns  and  recommendations. 
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•  Interview  the  analysis  implementation  staff  and  capture  their  concerns  and 
recommendations. 

•  Obtain  analysis  work  product  samples.  Access  to  organizations  and  people  should  be 
allowed  with  no  restrictions. 

•  Evaluate  analysis  activities  against  the  process  model. 

•  Document  the  study  results  and  recommendations. 

•  Prepare  and  present  preliminary  findings  and  recommendations. 

•  Prepare  and  deliver  the  final  findings  and  recommendations  report. 


6.6.3  Interviews 

The  appraisal  is  conducted  in  a  series  of  interviews,  of  two  types.  In  the  first  type,  the  team  conducts 
interviews  with  the  organization’s  management  with  regard  to  forthcoming  decisions  and  analysis 
support  needs.  Management  should  include  the  Director  and  his  Deputy,  the  heads  of  each  Group 
with  their  Deputies,  and  the  principal  staff  positions  of  Chief  Engineer  and  Lead  Scientist.  In  the  sec¬ 
ond  type,  the  appraisal  team  conducts  interviews  with  the  government  analysis  team  leaders  and 
sponsors  and  examines  documentation  to  discern  the  analysis  processes  actually  being  used  in  the 
organization.  These  interviews  are  modeled  on  the  techniques  used  for  CMMI  appraisals.  The  team 
examines  evidence  of  process  execution  without  an  evaluation  of  the  process  internals.  These  inter¬ 
views  are  used  to  discover: 

•  An  overview  of  analysis  tasks  that  the  interview  candidate  supports  and  their  areas  of 
responsibility 

•  Baseline  information  about  the  group  organization  and  activities 

•  Organization  charts,  processes,  capabilities  and  tools  for  analysis 

•  The  consumers  of  the  analysis  results  (internal  and  external) 

•  Governing  requirements,  policies,  directives,  etc.,  that  might  impact  approaches  and 
activities 

•  Summary  and  leading  current  group  activity  examples 

•  Schedules  and  definitions  of  future  analysis  applications 

•  Requirements  for  current  and  future  analyses  capabilities 

•  Known  issues,  shortfalls,  or  gaps 

•  Demonstrations,  if  available 

•  Areas  and  topics  of  future  analyses 

•  Understanding  of  analysis  process  oversight  by  management 

•  Process  documents  and  capability  descriptions 
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•  Accreditation  documents  along  with  the  supporting  Verification  and  Validation  docu¬ 
mentation  for  computer  models  and  simulations  in  use 

•  The  identity  of  the  Government  analysis  team  leaders  and  sponsors 

In  advance  of  the  interviews,  interview  questions  are  developed  that  focus  on  these  and  other  topics  of 
interest.  The  topics  covered  in  each  interview  are  selected  based  on  individual  areas  of  responsibility 
and  expertise.  Interviews  are  based  on  a  prepared  interview  questions  list  that  is  provided  to  the 
interviewees  ahead  of  time.  The  list  of  questions  is  constructed  to  assure  consistent  key  topics  cover¬ 
age  and  to  provide  a  record  of  what  was  asked.  Each  interview  session  should  have  a  designated 
recorder  to  capture  impromptu  and  follow-on  questions  that  were  not  scripted  ahead  of  time. 

At  the  end  of  the  interview  process,  results  are  analyzed,  and  follow-up  questions  developed,  if 
needed.  There  is  no  attribution  of  responses  to  interviewees  or  programs.  Once  all  responses  have 
been  received  and  analyzed,  the  findings  are  documented,  focused  on  each  of  the  key  interview  top¬ 
ics.  These  findings  are  then  briefed  back  to  the  interviewees  to  ensure  that  their  responses  are  accu¬ 
rately  stated. 


6.6.4  Appraisal  Report 

The  appraisal  team’s  report  to  the  sponsor  should  contain  both  findings  and  conclusions.  The  find¬ 
ings  are  facts  and  the  conclusions  are  the  team’s  judgments  based  on  the  facts.  Recommendations 
should  be  prioritized  both  in  terms  of  their  urgency  and  their  implementation  feasibility.  They  should 
also  be  divided  into  near-term  and  longer-term  recommendations.  As  much  as  possible,  the  report 
should  have  team  consensus.  However,  differences  of  opinion  can  be  documented.  Recommenda¬ 
tions  should  be  clearly  stated  and  should  be  specific  enough  to  indicate  what  needs  to  be  done.  The 
report  should  contain  only  recommendations  that  are  feasible  to  be  implemented.  If  multiple  organi¬ 
zations  are  being  appraised,  each  recommendation  should  show  organizational  responsibilities  for 
taking  the  action.  There  should  be  a  limited  number  of  recommendations  (4  or  5)  to  bring  the  right 
level  of  focus  (apply  groupings,  if  necessary).  The  report  should  be  presented  to  the  group  chiefs  and 
then  the  sponsor.  It  may  also  be  provided  to  other  stakeholders  who  may  be  affected  by  the  findings 
and  recommendations  if  the  sponsor  requests  this.  The  team  should  not  present  or  give  access  to  the 
report  to  any  others  without  the  appraisal  sponsor’s  permission.  For  a  possible  follow-up,  the  team 
can  consider  proposing  a  re-visit  at  a  later  date  to  see  what  impact  the  recommendations  have  had  and 
to  suggest  any  course  correction  on  the  direction  that  resulted  from  the  appraisal.  It  is  not  recom¬ 
mended  that  this  team  attempt  any  re -appraisal. 


6.7  Evidentiary  Artifacts  to  Corroborate  Conformity  to  Model 

Evidentiary  artifacts  may  take  form  of  hard  or  soft  copy  documentation,  verbal  corroboration  from  at 
least  two  sources,  or  other  data  as  deemed  appropriate  by  the  independent  appraisal  team.  This  sec¬ 
tion  delineates  some  typical  work  products  from  an  analysis  organization  that  provide  evidence  of  the 
existence  and  use  of  standard  processes  that  satisfy  the  ACMM.  These  artifacts  are  representative 
only  and  may  vary  widely  depending  on  the  organization’s  analysis  requirements.  These  are  defined 
at  the  Process  Area  Specific  Goal  level  and  include  evidence  from  each  of  the  specific  practices. 
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Two  evidentiary  artifacts  sources  might  include:  an  organizational  measurement  repository  that  con¬ 
tains  or  references  actual  measurement  data  and  related  information,  or  an  organizational  process 
asset  library  that  contains  information  used  to  store  and  make  available  process  assets  containing 
documents,  document  fragments,  and  process  implementation  aids.  This  process-related  documenta¬ 
tion  includes  policies,  defined  processes,  checklists,  lessons-leamed  documents,  templates,  standards, 
procedures,  plans,  and  training  materials.  Documents  include  both  paper  and  electronic  documents. 
(See  the  SEI  CMMI  specific  practices  for  typical  work  products  examples  in  related  practices.) 

The  following  items  are  examples  of  artifacts  that  were  identified  during  a  recent  appraisal  of  an 
analysis  organization  using  the  ACMM: 

•  Tasking  goals 

•  Regular  resource  meetings 

•  Analysis  plans 

•  Master  parameter  datasheets 

•  Defined  analysis  execution  for  cyclical  analyses 

•  Study  result  reviews 

•  Briefings  or  reports  containing  work  products 

•  Peer  reviews  of  models  with  cross-checking  with  other  models 

•  Experienced  team  leaders  judgment  of  SME  qualifications 

•  Use  of  Master  Schedule 

•  PMR  to  monitor  contractor  and  FFRDC  support 

•  Documentation  of  group  staff  meetings  to  monitor  study  status 

•  Stakeholder  involvement  in  studies 

•  Risk  identification  during  studies  planning  phase 

•  Inter-group  chiefs  one-on-one  meeting  minutes  to  ensure  that  analysis  support  meets  cur¬ 
rent  needs  and  anticipated  future  needs 

•  Reorganization  establishes  clearer  roles  and  responsibilities  within  analysis  organization 

•  Resource  allocation  tool  identifies  planned  support  level  per  task  and  to  track  planned 
staffing  assignments 

•  Appraisal  reviewed  by  intelligence  community  production  centers 

•  Informal  means  to  identify  and  control  configuration  items 

•  File  structure  for  storing  analysis  reports  on  networks 

•  Flexibility  to  “reach  back”  for  expertise 
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•  Instances  of  project-level  focus  on  process  or  product  improvement 

•  Guidelines  for  conducting  analyses 

•  Standard  process  for  data  calls  to  external  groups  to  get  correct  data  in  the  right  format 

•  Use  of  Popkin’s  System  Architect  tool  for  architecture  analysis 

•  Instances  of  new  staff  OJT 

6.8  Potential  Consequences  of  Not  Meeting  ACMM  Goals 

For  each  specific  goal,  the  effect  of  the  organization’s  processes  not  satisfying  the  ACMM’s  goals  has 
consequences  that  may  be  far  reaching.  The  potential  impacts  of  these  analysis  shortfalls  are  listed 
below.  The  only  real  remedy  is  to  develop  processes  to  conform  to  the  ACMM  process  areas. 

Analysis/Problem  Formulation  Goals 

•  Analysis  performed  may  fail  to  meet  executive  needs — wrong  analysis  executed. 

•  Analysis  might  be  delayed  while  the  analysis  team  struggles  with  formulating 
and  bounding  the  problem — analysis  may  not  be  timely. 

•  Analysis  may  be  more  costly  than  necessary  due  to  a  premature  analysis  team 
commitment;  that  is,  a  false  start  that  wastes  resources. 

•  Analysis  planning  may  be  inconsistent  with  executive  resource  commitments. 

•  Analysis  may  require  rework,  with  added  costs  and  time — resources  might  be 
wasted. 

Analysis  and  Data  Planning 

•  Measures  of  Merit  (MOM)  to  be  computed  may  not  be  identified,  the  algorithms 
to  be  used  may  not  be  specified,  and  the  data  to  be  gathered  may  not  be  known, 
resulting  in  the  wrong  or  incomplete  data  being  gathered. 

•  The  wrong  data  may  be  gathered 

•  Analysis  might  not  be  supported  by  certified  data. 

•  The  project  may  not  finish  before  the  resources  are  expended. 

•  Without  commitments  to  plans,  the  plans  may  not  be  carried  out. 

Analysis  Execution 

•  Analysis  results  may  not  be  accepted. 

•  Quantitative  results  may  not  be  generated. 


38 


•  Incomplete  or  erroneous  data  may  be  gathered. 

•  Incomplete  data  may  preclude  the  MOM  calculation. 

•  MOM  may  not  be  computed  correctly. 

•  Analysis  may  not  be  repeatable. 

Analysis  Resolution 

•  Decision  maker  may  not  understand  the  analysis  and  may  make  potentially 
wrong  decisions  or  may  fail  to  use  the  results  in  decision-making. 

•  Analysis  may  not  be  understood  by  stakeholders,  which  can  result  in  confusion 
and  analysis  rejection. 

•  Analysis  may  not  be  appropriate  for  the  problem  formulated. 

•  The  measures  of  merit  may  not  be  ranked  in  importance  to  the  decision  criteria 
with  a  potential  to  select  an  incorrect  alternative. 

•  The  decision  may  be  more  strongly  influenced  by  things  that  cannot  be  meas¬ 
ured.  If  the  incommensurables  and  other  factors  are  not  addressed,  a  faulty  deci¬ 
sion  might  be  made. 

Analysis  Reporting 

•  Documentation  may  be  incomplete,  which  limits  future  use  of  results  in  address¬ 
ing  similar  problems. 

•  Documentation  may  be  incomplete,  which  causes  reviewers  to  question 
conclusions. 

•  If  adequate  archives  are  not  maintained,  then  there  may  be  difficulty  in  answer¬ 
ing  future  questions  about  this  analysis 

Work  Product  Confirmation 

•  Selecting  the  wrong  work  products  for  confirmation  may  result  in  incomplete  or 
defective  confirmation,  which  may  result  in  defective  analyses  being  published. 

•  Incomplete  or  defective  peer  reviews  may  cause  defective  work  products  to  cor¬ 
rupt  the  analysis. 

•  Defects  that  are  not  discovered  until  analysis  resolution  may  cause  extensive 
rework. 

•  Work  products  may  not  meet  analysis  needs. 

•  Work  products,  especially  models  and  simulations,  may  not  perform  as  intended. 
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Model  or  Simulation  Corroboration 

•  Improper  models  or  simulations  may  not  reflect  the  real  world  and  lead  to  erro¬ 
neous  analysis  results  and  executive  decisions. 

•  Without  corroboration,  it  may  be  difficult  to  accredit  models  and  simulations. 

•  There  is  a  potential  that  the  analysis  can  be  refuted  or  not  accepted  if  there  is  a 
lack  of  confidence  that  the  supporting  models  and  simulations  are  correct 

SME  Qualification 

•  Lack  of  or  poor  SME  selection  criteria  could  lead  to  the  selection  of  unqualified 
SMEs,  which  could  mislead  the  analysis. 

•  Unqualified  SMEs  could  lead  to  erroneous  analysis  results  and  bad  decisions  by 
executives. 

•  If  the  necessary  qualifications  for  SME  are  not  specified,  the  expert  selected 
might  lack  the  desired  expertise. 

•  Analysis  might  not  be  repeatable  if  it  is  based  on  non-expert  opinions. 

Analysis  Monitor  and  Control 

•  Failure  to  monitor  with  a  resulting  loss  of  control  might  result  in  wasted 
resources. 

•  Analysis  may  not  be  accomplished  in  time  to  effect  a  decision. 

•  Resources  may  be  exhausted  before  the  analysis  is  completed. 

•  Critical  review  and  quality  assurance  steps  may  be  sacrificed  resulting  in  incom¬ 
plete  or  flawed  analysis. 

Analysis  Provider  Agreement 

•  Analysis  provider  selection  methods  might  lead  to  selecting  the  wrong  provider 
who  provides  poor  analysis  products,  resulting  in  bad  executive  decision  making. 

•  Without  proper  control,  an  analysis  provider  might  not  stick  to  agreement  and 
thereby  fail  to  meet  schedules  and  cause  delays  and  the  waste  of  resources. 

External  Stakeholder  Involvement  Management 

•  Lack  of  coordination  might  lead  to  duplication  of  effort  or  gaps  in  analysis  cov¬ 
erage,  resulting  in  wasted  resources. 

•  Poor  coordination  might  lead  to  a  misunderstanding  about  analysis  vision  result¬ 
ing  in  the  wrong  analysis. 
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•  Poor  communications  might  lead  to  frustration  and  poor  performance  by  the 
integrated  team. 

•  Necessary  analysis  data  from  external  stakeholders  may  not  be  available. 

Risk  Management 

•  Unidentified  risks  could  lead  to  rework. 

•  Unidentified  risks  might  cause  analysis  failure. 

•  Unmitigated  risks  could  result  in  surprises  and  unexpected  problems  with  an 
adverse  impact  on  analysis  cost,  schedule,  and  performance. 

•  An  improperly  structured  risk  management  process  might  lack  a  disciplined  envi¬ 
ronment  for  decision-making  and  the  efficient  use  of  program  resources. 

Internal  Analysis  Team  Management 

•  Without  a  defined  process  application,  the  analysis  might  flounder. 

•  Excessive  time  could  be  spent  to  determine  individual  roles,  responsibilities,  and 
relationships. 

•  Lack  of  a  shared  vision  for  the  analysis  could  mean  that  activities  are 
uncoordinated. 

•  Individual  activities  products  might  not  be  capable  of  being  integrated. 

•  Inappropriate  assignments  could  result  in  inefficient  or  ineffective  effort  by  team 
members. 

Quantitative  Analysis  Management 

•  Analysis  may  not  achieve  objectives. 

•  Analysis  may  not  meet  the  process-performance  objectives. 

•  Management  may  not  be  aware  of  analysis  progress  and  assets  consumption. 

•  Problems  with  the  analysis  might  not  be  identified  in  time  to  be  corrected;  e.g., 
variation  may  be  outside  the  control  bounds. 

Organizational  Process  Focus 

•  The  organization  may  develop  inappropriate  processes. 

•  Critical  process  areas  may  be  neglected  while  non-critical  processes  are 
developed. 
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•  Inadequate  processes  could  be  in  use,  which  could  result  in  inefficiencies  and 
ineffective  analysis. 

•  Workers  could  be  frustrated  with  the  lack  of  organizational  processes. 

•  Process  are  not  improved  and  opportunities  missed. 

Organizational  Process  Definition 

•  Process  assets  are  not  available  to  sustain  the  analysis. 

•  The  organization  could  be  unaware  of  process  measures  and  data  for  planning 
and  monitoring  the  analysis  process,  which  might  lead  to  schedule  slippage  and 
excessive  assets  consumption. 

•  Ad  hoc  and  inconsistent  processes  could  be  employed,  which  might  lead  to  inef¬ 
ficient  and  inconsistent  performance  . 

Organizational  Training 

•  Analysts  may  lack  the  skills  and  knowledge  to  execute  analysis. 

•  Effective  training  might  not  be  delivered,  which  could  result  in  wasted  resources. 

•  Training  records  may  not  be  maintained,  which  could  mean  that  job  assignments 
are  inappropriate. 

Organizational  Process  Performance 

•  The  organization  may  lack  the  data  to  manage  the  analysis  process,  which  could 
mean  that  the  analysis  exceeds  cost  and  the  schedule  slips. 

•  Lack  of  performance  models  may  mean  that  desired  performance  can  not  be 
stated,  which  could  mean  flawed  analysis  planning. 

Organizational  Innovation  and  Development 

•  Quality  objectives  derived  from  the  organization’s  business  plan  may  not  be 
achieved. 

•  Process-performance  objectives  derived  from  the  organization’s  business  plan 
may  not  be  achieved. 

•  The  organization’s  processes  and  technologies  may  not  be  measurably  improved. 

Measurement  and  Analysis 

•  Management  may  lack  data  to  make  informed  decisions. 
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•  Organization  may  not  be  able  to  objectively  measure  current  activities. 

•  Organizational  data  may  not  be  aligned  with  information  needs  and  objective; 
i.e.,  the  data  gathered  and  maintained  may  not  be  useful. 

Analysis  Process  and  Product  Quality  Assurance 

•  Management  and  staff  may  lack  objective  insight  into  process  and  associated 
work  product  quality. 

•  Flawed  analysis  is  published  and  presented  to  decision  makers. 

•  Even  good  analysis  may  lack  credibility. 

•  Incomplete  or  non-objective  reviews  may  result  in  defective  analysis  being 
published. 

Configuration  Management 

•  There  may  be  difficulty  in  Accreditation  of  Models  and  Simulations. 

•  Might  result  in  unreliable  analysis  results. 

•  Some  potential  for  unauthorized  changes  to  work  products  (data,  tools,  results, 
reports,  and  briefings). 

Establish  and  Maintain  Analytic  Capabilities 

•  It  is  possible  that  analytic  capabilities  will  degrade. 

•  Trained  analysts  may  not  be  available  for  future  analysis. 

•  Models  and  Simulations  may  not  be  capable  of  Accreditation. 

•  Databases  and  data  files  may  not  be  capable  of  Certification. 

Develop  Future  Analysis  Capabilities 

•  There  is  a  possibility  that  the  organization  will  spend  limited  resources  on  creating  or 
improving  capabilities  for  which  there  is  little  need. 

•  There  is  a  possibility  that  the  organization  will  lack  the  necessary  capabilities  to 
perform  appropriate  future  analysis. 

•  There  is  a  possibility  that  future  analysis  will  be  driven  by  stale  capabilities  vice 
decision  maker  needs. 

Causal  Analysis  and  Resolution 


43 


•  Defects  could  be  repeated  in  future  work. 

•  Defect  detection  might  be  casually  treated  if  defect  causes  are  not  analyzed  and 
addressed. 


6.9  Potential  Benefits  of  Meeting  ACMM  Goals 

If  the  ACMM  goals  are  satisfied,  there  are  potential  benefits  to  the  organization’s  process  improve¬ 
ments  and  its  ability  to  perform  quality  analyses,  resulting  in  mission  success  for  the  analysis  organi¬ 
zation.  Helping  to  establish  and  meet  these  goals  is  one  of  the  main  purposes  of  creating  and  using 
the  ACMM. 


A  listing  of  some  of  the  expected  benefits  of  using  this  model  include: 

•  A  better  understanding  of  the  role  that  analysis  plays  in  the  organization’s  processes 

•  Appraisal  of  the  degree  of  analysis  collaboration  and  coordination  within  the  organization 

•  Recommendations  for  monitoring,  controlling,  and  reporting  analyses 

•  Recommended  analysis  processes  with  criterion  and  documentation  requirements,  if 
needed 

•  Recommended  process  for  future  analyses  development 

•  A  current  tool  suite  appraisal  and  its  suitability  for  future  analyses 

•  Independent  support  for  analysis  tool  investment 

•  Following  well-defined  analysis  processes  improves  customer  satisfaction,  resource  effi¬ 
ciency,  analysis  quality  and  timeliness,  results  preservation,  and  support  to  decision 
makers. 

•  Managing,  monitoring,  and  controlling  analysis  processes  helps  to  ensure  effective  com¬ 
munication,  helps  studies  utilize  organizational  resources  appropriately,  helps  reduce  task 
risks,  and  helps  assure  that  objectives  are  met. 

•  Metrics,  Quality  Assurance,  and  Configuration  Management  help  executives  make 
informed  decisions  on  activities  and  trends.  They  also  provide  visibility  and  feedback  on 
the  processes  and  work  products  and  enable  the  storage  and  retrieval  of  analysis  work 
products  and  final  results. 

•  Clarifying  the  organization’s  mission  roles  and  responsibilities  helps  ensure  that  it  is  well 
suited  for  satisfying  the  current  and  future  customer  analysis  needs  and  to  apply  limited 
resources  appropriately. 

•  Managing  processes  properly  provides  the  means  for  consistently  identifying  and  acting 
on  improvement  opportunities;  enhance  quality;  and  use  training  resources  effectively. 

•  Proper  analysis  formulation  can  improve  meeting  customer  expectations  in  a  timely  man¬ 
ner  and  prevent  inefficient  use  of  resources. 
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•  Analysis  planning  helps  ensure  that  analysis  approach  is  properly  defined  to  meet  objec¬ 
tives,  tools  are  correctly  applied,  and  the  appropriate  data  is  utilized. 

•  Proper  analysis  execution  and  review  increases  the  acceptability  and  analysis  quality. 

•  Creating,  reviewing,  and  archiving  analysis  documents  captures  the  work  performed  by 
the  organization,  ensures  it  is  accurately  represented,  and  preserves  it  for  future  reference. 

•  Properly  corroborated  models  and  simulations  can  lead  to  higher  quality  analysis  results 
and  better  support  to  decision  makers. 

•  Utilizing  appropriately  qualified  SME  support  can  provide  the  organization  with  the 
expertise  needed  to  produce  high-quality  results  in  support  of  decision  makers. 

•  Analysis  monitor  and  control  helps  ensure  that  analyses  are  accomplished  as  planned — in 
a  timely  manner — and  resources  are  not  wasted. 

•  External  stakeholders  involvement  helps  to  ensure  that  analyses  results  are  in  keeping 
with  external  stakeholder  expectations,  limits  duplication  of  efforts,  and  avoids  gaps  in 
analysis. 

•  Risk  management  processes  help  identify  and  mitigate  issues  early  to  minimize  failures 
and  avoid  rework. 

•  Analysis  process  management  improves  the  quality  and  study  results  effectiveness  by 
leveraging  from  expertise  across  the  organization. 

•  Collaboration  and  communications  improve  communications  quality  and  clarifies  roles 
and  responsibilities. 

•  Established  measurement  objectives,  metrics,  and  reporting  capability  helps  measure  cur¬ 
rent  activities,  track  trends,  project  future  needs,  and  make  informed  management 
decisions. 

•  Quality  assurance  supports  the  delivery  of  high-quality  analysis  by  providing  visibility 
and  feedback  on  the  processes  and  work  products  throughout  the  project  life  and  across 
the  organization. 

•  Configuration  management  enables  analysis  reports  and  supporting  information  storage 
and  retrieval  so  that  analysis  integrity  and  history  can  be  maintained,  study  approach  and 
results  reused,  and  models  and  simulation  accredited. 

•  Establishing  and  maintaining  analysis  capabilities  helps  prevent  staff  analytic  capabilities 
degradation  and  tools  effectiveness. 

•  Identifying  future  analysis  needs  based  on  stakeholder  and  customer  requirements  helps 
to  ensure  that  the  organization  is  well  suited  to  carry  out  the  customer  analysis  needs  and 
to  spend  limited  resources  efficiently. 

Continuous  analysis  process  improvement  identifies  opportunities  to  improve  efficiency,  enhance 
effectiveness,  evolve  best  practices,  fix  inadequate  processes,  eliminate  inappropriate  processes, 
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reduce  time  spent  on  non-critical  processes,  make  the  entire  organization  and  each  individual  an  inte¬ 
gral  part  of  improvement  operations,  and  improve  analysis. 

Establishing  and  maintaining  useable  and  effective  standard  processes  make  process  assets  readily 
available  to  sustain  analysis  and  control  the  use  of  inconsistent  processes,  which  might  lead  to  ineffi¬ 
cient  and  inconsistent  performance. 

Developing  staff  skills  and  knowledge  in  key  areas  will  enhance  the  analysis  results  quality,  assure 
effective  use  of  training  resources,  minimize  ramp-up  time  for  new  staff,  and  avoid  false  starts  due  to 
lack  of  process  awareness. 
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7.  Recommendations 


The  ACMM  is  recommended  to  analysis  organizations  interested  in  improving  the  reliability  and 
quality  of  their  studies.  It  is  also  recommended  to  the  purchasers  of  operations  research  and  system 
analysis  who  are  often  asked  to  take  the  conclusions  and  recommendations  of  a  study  on  faith  alone. 
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Appendix — Analysis  CMM  Graphics 


The  following  charts  provide  a  graphical  representation  of  the  ACMM  process  areas,  specific  goals, 
specific  practices,  and  the  CMMI  generic  goals  and  practices.  These  charts  may  aid  in  understanding 
the  overall  organization  and  structure  of  the  model. 
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